Skip to content

feat: serve dig.getManifest locally from the embedded public manifest - #1

Merged
MichaelTaylor3d merged 1 commit into
mainfrom
feat/get-manifest-176c
Jul 4, 2026
Merged

feat: serve dig.getManifest locally from the embedded public manifest#1
MichaelTaylor3d merged 1 commit into
mainfrom
feat/get-manifest-176c

Conversation

@MichaelTaylor3d

Copy link
Copy Markdown
Contributor

Summary

#176 Phase C: dig.getManifest moves from a blind-passthrough alias to a LOCAL read, resolving the normalized PublicManifest (.dig data-section id 13, Phase A — already merged in digstore's store-libs) embedded in a held capsule's compiled module.

  • dig-node-core: new Node::get_manifest + read_public_manifest_blocking (runs on spawn_blocking, mirroring serve_local_blocking). Params {store_id, root} (a capsule identifier — storeId:rootHash, matching dig.getAvailability/dig.fetchRange's shape); no retrieval_key needed since the manifest is PUBLIC, unencrypted data.
    • Module held + manifest present → result = the manifest JSON ({schema_version, entries:[...]}, byte-identical to PublicManifest::to_json).
    • Module held + no manifest section (an older .dig, or a private store) → result: nullnever an error (store-format §5.1 backwards-compatibility rule).
    • Capsule not held locally at all → -32004 (same code dig.fetchRange uses on a miss).
    • Malformed store_id/root-32602 before any filesystem access.
  • dig.stage now sets FinalizeOptions.include_public_manifest for PUBLIC stores (matching the digstore CLI's own commit rule) — required because the digstore rev bump below adds this as a new field on that struct.
  • Bumped the pinned digstore rev (dig-node-coreanddig-wallet, kept in sync per the existing "bump together" contract) to the tip carrying the public-manifest format extension (digstore-core/-store/-compiler/-stage etc., no digstore-guest changes — the vendored guest wasm did not need rebuilding). Added digstore-compiler as a direct dependency of dig-node-core for extract_data_section_blob (a pure wasm-binary-format parse — no wasmtime instantiation needed to read this public section).
  • dig-node-service: meta.rs catalogue entry flipped served: "passthrough""local" with an updated summary; doc comments in server.rs/rpc.rs updated to match (a clarifying comment was added to rpc.rs explaining Route::Passthrough there is a request-normalization classification, independent of local-vs-relay serving).
  • SPEC.md/README.md/USER_JOURNEY.md updated to document the new local behavior (including a new SPEC §5.5.1) instead of describing it as a blind-passthrough alias.

Version bump

dig-node-core0.1.00.2.0 and dig-node-service0.3.00.4.0minor: a new capability (dig.getManifest is now resolved by the node instead of always relaying to the upstream), fully backwards compatible (existing callers of the method still get an answer; everything else is unchanged).

Tests (TDD — written before the implementation)

  • dig_get_manifest_returns_embedded_manifest_json_when_present
  • dig_get_manifest_returns_null_when_section_absent
  • dig_get_manifest_reports_unavailable_when_capsule_not_held
  • dig_get_manifest_rejects_malformed_params_without_touching_disk

Both manifest-presence tests build a real compiled .dig module via digstore_stage::stage_and_compile (not a mock) so the data-section extraction + decode is exercised end-to-end. The pre-existing passthrough_alias_methods_are_method_not_found_on_the_node test was updated to drop dig.getManifest from the passthrough-alias list, and the openrpc_drift_guard suite (which dispatches every catalogued method through the real handle_rpc) was re-verified green against the new served: "local" classification.

How verified

  • cargo fmt --all -- --check — clean.
  • cargo clippy --workspace --all-targets --all-features -- -D warnings — zero warnings.
  • cargo test --workspace — all green: dig-node-core 193 passed, dig-node-service 92 passed (incl. the 5 openrpc_drift_guard tests + the 25 server.rs integration tests), dig-wallet 71 passed, dig-runtime 3 passed. Zero failures.
  • Confirmed the digstore rev bump resolves to a single coherent version of every digstore-* crate across the whole workspace (dig-wallet had its own independent, now-stale pin — bumped it too to keep the documented "same rev as the node" invariant).

Note: this repo's current required PR checks are "Lint commit messages" (commitlint) and "Check version increment" only — there is no ci.yml running cargo test/clippy/fmt as a required gate yet. I ran all of the above locally per the ecosystem-wide contract; adding a dedicated CI workflow for it would be a separate, repo-wide infra task.

#176 Phase C: dig.getManifest moves from a blind-passthrough alias to a
LOCAL read, resolving the normalized PublicManifest (.dig data-section id
13, Phase A) embedded in a held capsule's compiled module.
- dig-node-core: new Node::get_manifest + read_public_manifest_blocking
(spawn_blocking, mirrors serve_local_blocking). Params {store_id, root}
(a capsule identifier, no retrieval_key needed - the manifest is public,
unencrypted data). A held module with no manifest section (an older
.dig, or a private store) returns `result: null`, never an error
(store-format backwards-compatibility rule); a capsule not held at all
reports -32004, matching dig.fetchRange's miss code.
- dig.stage now sets FinalizeOptions.include_public_manifest for PUBLIC
stores, matching the digstore CLI's own commit rule (a new required
field added by the digstore rev bump below).
- Bumped the pinned digstore rev (dig-node-core + dig-wallet, kept in
sync) to the tip carrying the public-manifest format extension, and
added digstore-compiler as a direct dependency for
extract_data_section_blob (already a transitive dep via digstore-stage).
- dig-node-service: meta.rs catalogue entry flipped to served: "local"
with an updated summary; doc comments in server.rs/rpc.rs updated to
match.
- SPEC.md/README.md/USER_JOURNEY.md updated to describe the new local
behavior instead of the old passthrough alias.
- Tests: dig_get_manifest_returns_embedded_manifest_json_when_present,
dig_get_manifest_returns_null_when_section_absent,
dig_get_manifest_reports_unavailable_when_capsule_not_held,
dig_get_manifest_rejects_malformed_params_without_touching_disk. The
existing passthrough_alias_methods_are_method_not_found_on_the_node
test and the openrpc_drift_guard suite were updated/re-verified against
the new classification.
Verified: cargo fmt --all -- --check, cargo clippy --workspace
--all-targets --all-features -- -D warnings, and cargo test --workspace
all green (dig-node-core 193, dig-node-service 92, dig-wallet 71,
dig-runtime 3 - zero failures).
@MichaelTaylor3d
MichaelTaylor3d merged commit 0d70f4a into mainJul 4, 2026
2 checks passed
@MichaelTaylor3d
MichaelTaylor3d deleted the feat/get-manifest-176c branch July 4, 2026 10:00
MichaelTaylor3d added a commit that referenced this pull request Jul 10, 2026
…#1)
#176 Phase C: dig.getManifest moves from a blind-passthrough alias to a
LOCAL read, resolving the normalized PublicManifest (.dig data-section id
13, Phase A) embedded in a held capsule's compiled module.
- dig-node-core: new Node::get_manifest + read_public_manifest_blocking
(spawn_blocking, mirrors serve_local_blocking). Params {store_id, root}
(a capsule identifier, no retrieval_key needed - the manifest is public,
unencrypted data). A held module with no manifest section (an older
.dig, or a private store) returns `result: null`, never an error
(store-format backwards-compatibility rule); a capsule not held at all
reports -32004, matching dig.fetchRange's miss code.
- dig.stage now sets FinalizeOptions.include_public_manifest for PUBLIC
stores, matching the digstore CLI's own commit rule (a new required
field added by the digstore rev bump below).
- Bumped the pinned digstore rev (dig-node-core + dig-wallet, kept in
sync) to the tip carrying the public-manifest format extension, and
added digstore-compiler as a direct dependency for
extract_data_section_blob (already a transitive dep via digstore-stage).
- dig-node-service: meta.rs catalogue entry flipped to served: "local"
with an updated summary; doc comments in server.rs/rpc.rs updated to
match.
- SPEC.md/README.md/USER_JOURNEY.md updated to describe the new local
behavior instead of the old passthrough alias.
- Tests: dig_get_manifest_returns_embedded_manifest_json_when_present,
dig_get_manifest_returns_null_when_section_absent,
dig_get_manifest_reports_unavailable_when_capsule_not_held,
dig_get_manifest_rejects_malformed_params_without_touching_disk. The
existing passthrough_alias_methods_are_method_not_found_on_the_node
test and the openrpc_drift_guard suite were updated/re-verified against
the new classification.
Verified: cargo fmt --all -- --check, cargo clippy --workspace
--all-targets --all-features -- -D warnings, and cargo test --workspace
all green (dig-node-core 193, dig-node-service 92, dig-wallet 71,
dig-runtime 3 - zero failures).
Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d added a commit that referenced this pull request Jul 10, 2026
…#1)
#176 Phase C: dig.getManifest moves from a blind-passthrough alias to a
LOCAL read, resolving the normalized PublicManifest (.dig data-section id
13, Phase A) embedded in a held capsule's compiled module.
- dig-node-core: new Node::get_manifest + read_public_manifest_blocking
(spawn_blocking, mirrors serve_local_blocking). Params {store_id, root}
(a capsule identifier, no retrieval_key needed - the manifest is public,
unencrypted data). A held module with no manifest section (an older
.dig, or a private store) returns `result: null`, never an error
(store-format backwards-compatibility rule); a capsule not held at all
reports -32004, matching dig.fetchRange's miss code.
- dig.stage now sets FinalizeOptions.include_public_manifest for PUBLIC
stores, matching the digstore CLI's own commit rule (a new required
field added by the digstore rev bump below).
- Bumped the pinned digstore rev (dig-node-core + dig-wallet, kept in
sync) to the tip carrying the public-manifest format extension, and
added digstore-compiler as a direct dependency for
extract_data_section_blob (already a transitive dep via digstore-stage).
- dig-node-service: meta.rs catalogue entry flipped to served: "local"
with an updated summary; doc comments in server.rs/rpc.rs updated to
match.
- SPEC.md/README.md/USER_JOURNEY.md updated to describe the new local
behavior instead of the old passthrough alias.
- Tests: dig_get_manifest_returns_embedded_manifest_json_when_present,
dig_get_manifest_returns_null_when_section_absent,
dig_get_manifest_reports_unavailable_when_capsule_not_held,
dig_get_manifest_rejects_malformed_params_without_touching_disk. The
existing passthrough_alias_methods_are_method_not_found_on_the_node
test and the openrpc_drift_guard suite were updated/re-verified against
the new classification.
Verified: cargo fmt --all -- --check, cargo clippy --workspace
--all-targets --all-features -- -D warnings, and cargo test --workspace
all green (dig-node-core 193, dig-node-service 92, dig-wallet 71,
dig-runtime 3 - zero failures).
Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d added a commit that referenced this pull request Aug 2, 2026
)
* feat(dig-node-core): add pure tier-0 knapsack selector (#1988)
Add tier0_selector: a pure, deterministic module that picks which
speculative-precache candidates are worth keeping under a small
sub-budget of DIG_NODE_CACHE_CAP, reusing relevance()/RelevanceValue/
should_displace from the child-1 relevance module rather than
reimplementing scoring or hysteresis.
- Candidate{size_bytes, relevance} + select_within_budget: O(n log n)
greedy by value-density (relevance/size), never exceeding the given
budget. Deliberately greedy, not DP: at GiB-scale budgets a DP table
is disproportionate, and greedy under-fills by at most one
candidate's size.
- tier0_budget_bytes: TIER0_BUDGET_FRACTION (0.10) of the whole cache
cap, pure arithmetic over a caller-supplied cap (no I/O in this
module).
- should_displace_tier0: thin named wrapper over should_displace with
the selector's own DEFAULT_HYSTERESIS_MARGIN, so a marginally-better
candidate cannot displace an incumbent and thrash fetch/evict/refetch.
Also fixes two comment-accuracy nits left over from child #1 in
relevance.rs: the xor-weight-dominance comment now states it exceeds
only the ATTACKER-GAMEABLE secondaries (scarcity+demand+recency=0.85),
excluding the operator-controlled pin/pin_adjacent bonuses; the
proximity-map comment now says `hi128 / u128::MAX` to match what the
code actually divides by. No behaviour change.
Bumps: crates/dig-node-core 0.27.0 -> 0.28.0 (MINOR, new export) and
the root workspace [workspace.package].version (the released dig-node
binary) 0.76.0 -> 0.77.0 (MINOR), with Cargo.lock regenerated to match.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(dig-node): spec the tier-0 knapsack selector + fix rustfmt (#1988)
Add SPEC.md §7.10b normativley describing tier0_selector's contract
(the 10% sub-budget, greedy-by-density selection, and the hysteresis
reuse of should_displace) alongside §7.10a's relevance model. Also
applies the rustfmt ordering rustfmt itself wants for the new
`pub mod tier0_selector;` declaration in lib.rs.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
MichaelTaylor3d pushed a commit that referenced this pull request Aug 2, 2026
…tion (#1987)
Produce the tier-0 precache CANDIDATE SET (epic #1934, child 2/7): random
keyspace sampling picks which neighbourhoods to probe, and a pure multi-peer
reconciliation admits a content key only on an M-of-N distinct-peer quorum,
deriving each key's provider count from the lower median (never a liar's
inflated max). Feeds relevance()'s RelevanceInputs; does not score, select,
or fetch (children #1/#3/#4).
- reconcile(): per-peer dedup then quorum + median aggregate (pure, no I/O)
- sample_keyspace_points(): deterministic under an injected SplitMix64 RNG
- sample_candidates(): async composition over a NeighbourhoodProbe seam
- SPEC.md §7.10c documents the sampling + quorum contract (normative)
Bumps dig-node-core 0.28.0 -> 0.29.0 and workspace 0.77.0 -> 0.78.0 (MINOR).
Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d added a commit that referenced this pull request Aug 2, 2026
…tion (#153)
* feat(dig-node): scaffold DHT candidate sampling (#1987)
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(dig-node): DHT candidate sampling + anti-Sybil quorum reconciliation (#1987)
Produce the tier-0 precache CANDIDATE SET (epic #1934, child 2/7): random
keyspace sampling picks which neighbourhoods to probe, and a pure multi-peer
reconciliation admits a content key only on an M-of-N distinct-peer quorum,
deriving each key's provider count from the lower median (never a liar's
inflated max). Feeds relevance()'s RelevanceInputs; does not score, select,
or fetch (children #1/#3/#4).
- reconcile(): per-peer dedup then quorum + median aggregate (pure, no I/O)
- sample_keyspace_points(): deterministic under an injected SplitMix64 RNG
- sample_candidates(): async composition over a NeighbourhoodProbe seam
- SPEC.md §7.10c documents the sampling + quorum contract (normative)
Bumps dig-node-core 0.28.0 -> 0.29.0 and workspace 0.77.0 -> 0.78.0 (MINOR).
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
MichaelTaylor3d added a commit that referenced this pull request Aug 6, 2026
…tadata (#185)
Closes dig_ecosystem#2145. Hardens two DoS gaps on the anonymous, internet-reachable public-manifest read path. The ticket's fix#1 (the manifest-memo byte budget: MANIFEST_MEMO_MAX_BYTES=32 MiB + MANIFEST_ENTRY_MAX_BYTES=4 MiB + LRU) was already on main; this closes the two remaining gaps:
1. cache.clear now drains the process-lifetime manifest memo (it previously left it intact, so an operator couldn't reclaim it).
2. dig.getMetadata gets a 3 MiB response ceiling. It was the only public-tier read with no response cap (~200 B in -> up to ~100 MB out). A whole JSON object can't be windowed, so an oversized section is refused with a bounded -32015 METADATA_TOO_LARGE. read_metadata_manifest_json returns the rendered Arc<str> so get_metadata checks the raw length BEFORE parsing, so the oversized body is never parsed + re-serialized. Normal metadata serves byte-identically.
Both gates PASS: loop-reviewer (correctness) + loop-security (the getMetadata ceiling is a hard pre-parse bound on every render path, no bypass; cache.clear is control-token-gated, not anonymous-reachable; every arm fails closed). The -32015 error is catalogued in docs.dig.net (companion PR #66). Non-gating follow-ups: #2217 (error-code drift), #2218 (gitnexus runner), #2219 (encoded-length pre-filter).
root [workspace.package].version 0.99.1 -> 0.99.2 (patch, fix).
Closes #2145
Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@MichaelTaylor3d