pagestore: exercise image layer publication crashes

.github/workflows/pagestore-test.yml

@@ -76,13 +76,14 @@ jobs:
  cc -O2 -Wall -Wextra -Werror -o pagestore_daemon pagestore_daemon.c pagestore_core.c pagestore_fault.c storage_posix.c pagestore_layer.c pagestore_layer_store.c pagestore_manifest.c pagestore_memtable.c pagestore_pgcache.c pagestore_prune.c pagestore_retention.c pagestore_wal_store.c pagestore_wal_segment.c pagestore_walidx_prune.c pagestore_walidx_snapshot.c pagestore_forkmeta_prune.c pagestore_forkmeta_snapshot.c -lrt -lpthread
  cc -O2 -Wall -Wextra -Werror -o pagestore_inspect pagestore_inspect.c -lrt
  cc -O2 -Wall -Wextra -Werror -o pagestore_test pagestore_test.c -lrt
- cc -O2 -Wall -Wextra -Werror -o pagestore_layer_test pagestore_layer_test.c pagestore_layer.c pagestore_layer_store.c -lrt
- cc -O2 -Wall -Wextra -Werror -o pagestore_layer_store_test pagestore_layer_store_test.c pagestore_layer_store.c pagestore_layer.c -lrt
+ cc -O2 -Wall -Wextra -Werror -o pagestore_layer_test pagestore_layer_test.c pagestore_layer.c pagestore_layer_store.c pagestore_fault.c -lrt -pthread
+ cc -O2 -Wall -Wextra -Werror -o pagestore_layer_store_test pagestore_layer_store_test.c pagestore_layer_store.c pagestore_layer.c pagestore_fault.c -lrt -pthread
+ cc -O2 -Wall -Wextra -Werror -o pagestore_layer_crash_client pagestore_layer_crash_client.c -lrt -pthread
  cc -O2 -Wall -Wextra -Werror -o pagestore_tiering_test pagestore_tiering_test.c pagestore_core.c pagestore_fault.c storage_posix.c pagestore_layer.c pagestore_layer_store.c pagestore_manifest.c pagestore_memtable.c pagestore_pgcache.c pagestore_prune.c pagestore_retention.c pagestore_wal_store.c pagestore_wal_segment.c pagestore_walidx_prune.c pagestore_walidx_snapshot.c pagestore_forkmeta_prune.c pagestore_forkmeta_snapshot.c -lrt -lpthread
  cc -O2 -Wall -Wextra -Werror -o pagestore_forkmeta_cutover_test pagestore_forkmeta_cutover_test.c pagestore_core.c pagestore_fault.c storage_posix.c pagestore_layer.c pagestore_layer_store.c pagestore_manifest.c pagestore_memtable.c pagestore_pgcache.c pagestore_prune.c pagestore_retention.c pagestore_wal_store.c pagestore_wal_segment.c pagestore_walidx_prune.c pagestore_walidx_snapshot.c pagestore_forkmeta_prune.c pagestore_forkmeta_snapshot.c -lrt -lpthread
  cc -O2 -Wall -Wextra -Werror -o pagestore_forkmeta_crash_matrix_test pagestore_forkmeta_crash_matrix_test.c pagestore_core.c pagestore_fault.c storage_posix.c pagestore_layer.c pagestore_layer_store.c pagestore_manifest.c pagestore_memtable.c pagestore_pgcache.c pagestore_prune.c pagestore_retention.c pagestore_wal_store.c pagestore_wal_segment.c pagestore_walidx_prune.c pagestore_walidx_snapshot.c pagestore_forkmeta_prune.c pagestore_forkmeta_snapshot.c -lrt -lpthread
  cc -O2 -Wall -Wextra -Werror -o pagestore_pgcache_test pagestore_pgcache_test.c pagestore_pgcache.c -lrt -lpthread
- cc -O2 -Wall -Wextra -Werror -o pagestore_manifest_test pagestore_manifest_test.c pagestore_manifest.c pagestore_layer.c pagestore_layer_store.c -lrt
+ cc -O2 -Wall -Wextra -Werror -o pagestore_manifest_test pagestore_manifest_test.c pagestore_manifest.c pagestore_layer.c pagestore_layer_store.c pagestore_fault.c -lrt -pthread
  cc -O2 -Wall -Wextra -Werror -o pagestore_retention_test pagestore_retention_test.c pagestore_retention.c -lrt -lpthread
  cc -O2 -Wall -Wextra -Werror -o pagestore_gc_test pagestore_gc_test.c pagestore_core.c pagestore_fault.c storage_posix.c pagestore_layer.c pagestore_layer_store.c pagestore_manifest.c pagestore_memtable.c pagestore_pgcache.c pagestore_prune.c pagestore_retention.c pagestore_wal_store.c pagestore_wal_segment.c pagestore_walidx_prune.c pagestore_walidx_snapshot.c pagestore_forkmeta_prune.c pagestore_forkmeta_snapshot.c -lrt -lpthread
  cc -O2 -Wall -Wextra -Werror -o pagestore_walidx_prune_test pagestore_walidx_prune_test.c pagestore_walidx_prune.c -lrt
@@ -109,6 +110,22 @@ jobs:
  --daemon-binary ./pagestore_daemon \
  --inspect-binary ./pagestore_inspect
 
+ - name: Run image-layer H1 crash scenarios
+ working-directory: contrib/pagestore
+ run: |
+ for scenario in \
+ image_layer_after_create \
+ image_layer_after_write \
+ image_layer_after_seal \
+ image_layer_after_manifest_add; do
+ python3 harness/pagestore_harness.py \
+ --capabilities harness/capabilities.json \
+ --daemon-fault-recovery "harness/scenarios/${scenario}.jsonl" \
+ --daemon-binary ./pagestore_daemon \
+ --layer-client-binary ./pagestore_layer_crash_client \
+ --inspect-binary ./pagestore_inspect
+ done
+
  - name: Run image-layer unit test
  working-directory: contrib/pagestore
  run: ./pagestore_layer_test

contrib/pagestore/MVP_COMPLETION_PLAN.md

@@ -783,8 +783,9 @@ so the SPDK frontend remains runnable without that POSIX-only mailbox.
 The first composed H1 materializer slice is now implemented: the two
 restartpoint plans pause the checkpointer child after relation-page sync/before
 marker write and after marker sync, then stop and recover the whole
-materializer. The remaining branch, layer, reclaim, and GC H1 families remain
-outstanding.
+materializer. The prepared-receipt/service-restore branch slice and the POSIX
+image-layer create/write/seal/manifest-ADD publication slice are also covered.
+Branch bootstrap/install, manifest replacement, reclaim, and GC H1 cases remain.
 
 Deliverables:
 
@@ -810,8 +811,10 @@ Expected scope: one or two PRs.
 
 ### H1. Compose process-level crash scenarios
 
-Status: **materializer replay/restartpoint slice implemented; branch, layer,
-reclaim, and GC cases remain and depend on H0/R2-R5**.
+Status: **materializer replay/restartpoint, branch prepared-receipt/service-
+restore, and POSIX image-layer publication slices implemented; branch
+bootstrap/install, manifest replacement, reclaim, and GC cases remain and
+depend on H0/R2-R5**.
 
 Required scenario families:
 

contrib/pagestore/MVP_STATUS.md

@@ -42,7 +42,7 @@ new daemon's zeroing/recovery window.
 | Area | Status | Current proof |
 |---|---|---|
 | Page ingest and copy-on-write reads | Implemented | standalone and PostgreSQL integration suites |
-| Image-layer path | Functional mechanisms implemented; phases 2–3 partial | manifest/compaction/segment-GC restart tests; sparse indexes and layer-block cache invalidation remain |
+| Image-layer path | Functional mechanisms implemented; H1 POSIX publication crash slice covered; phases 2–3 partial | manifest/compaction/segment-GC restart tests plus create/write/seal/manifest-ADD crash recovery scenarios with sentinel/LSN and idempotent restart checks; sparse indexes and layer-block cache invalidation remain |
 | Filesystem object tier | Upload done; cache/GC operations partial | download, eviction, refresh, and remote-delete tests; cache policy and orphan reconciliation remain |
 | Materialized-page cache | Basic version cache implemented; phase partial | bounded cache/invalidation tests; cost-aware admission and integrated redo avoidance remain |
 | WAL shipping and ancestry-aware WAL reads | Immutable 1 MiB segments integrated for sealed prefixes; the flat-log copy of every complete sealed record is reclaimed, while the flat log remains migration/tail authority | chunk assembly, reopen, ancestry, and WAL segment/store tests |
@@ -78,6 +78,18 @@ The existing CI proves both focused subsystem paths and the composed contract:
 - `branch_boot_test.sh` proves an independent branch compute can boot, preserve
  fork-point visibility, and write on its own timeline.
 
+The H1 image-layer crash slice is limited to POSIX local layers and process
+abort. Its four ordered stages cover canonical file creation, file writes
+before seal, sealed layer data, and durable `layers.manifest` ADD publication;
+the write stage does not claim power-loss durability. The composed
+harness keeps the pre-recovery physical snapshot, then checks a sentinel
+page/LSN, the expected manifest state, and one additional restart for
+idempotence. A crash after ADD but before its flush watermark conservatively
+retains the durable layer and republishes segment-backed coverage once; the
+intervening clean shutdown compacts that conservative duplicate, and the second
+restart proves convergence to one layer. The slice does not delete crash
+orphans; cross-process ownership is required before adding that policy.
+
 ## MVP gates
 
 ### 1. One composed golden scenario -- implemented
@@ -308,12 +320,13 @@ mutations and its snapshot maintenance can be forced below the geometric
 trigger. All three controllers are POSIX-only; forkmeta does not claim the
 remaining R6 queue-bound soak/tuning work.
 
-### 5. Composed crash and format-compatibility coverage -- remaining
+### 5. Composed crash and format-compatibility coverage -- partial
 
-Focused crash-safety tests exist, while the declarative harness remains partial.
+The POSIX image-layer publication slice is now covered by the declarative
+harness. Other crash boundaries remain outside this slice.
 Before declaring the MVP repeatable, add process-level fault scenarios around
-materializer progress, branch prepare/install, manifest/layer publication, and
-retention GC, plus a persisted-format fixture for restart/upgrade compatibility.
+branch bootstrap/install, manifest replacement, and retention/reclaim/GC, plus
+a persisted-format fixture for restart/upgrade compatibility.
 
 ## Recommended sequence
 

contrib/pagestore/harness/capabilities.json

@@ -13,11 +13,12 @@
  "io_unit": 262144
  },
  "daemon_fault_smoke": {
- "operations": ["crash", "set_fault", "release_fault"],
+ "operations": ["crash", "set_fault", "release_fault", "layer_seed"],
  "constraints": {
  "crash": {},
  "set_fault": {},
- "release_fault": {"target": ["store"]}
+ "release_fault": {"target": ["store"]},
+ "layer_seed": {"target": ["store"]}
  },
  "protocol_version": 45,
  "page_size": 8192,
@@ -71,6 +72,7 @@
  "install_reader",
  "restart",
  "crash",
+ "layer_seed",
  "advance",
  "assert",
  "materializer_fault",