diff --git a/CLAUDE.md b/CLAUDE.md index b1f97be..f40dad8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -70,7 +70,7 @@ When debugging container behavior, the order is: image `/etc/{rc,fstab,environme ## Active design proposals - **`doc/zfs.md`** — optional ZFS storage backend (`ENROOT_STORAGE_BACKEND=zfs`). Replaces `unsquashfs`-per-create with extract-once-then-`zfs clone`. Adds a `.zfs` (zfs send stream) image format and a `zfs://host/NAME` transport scheme alongside today's `.sqsh`. Introduces a shared template cache with a live/warm/cold lifecycle (knobs: `ENROOT_TEMPLATE_WARM_SECONDS`, `ENROOT_TEMPLATE_PRESSURE_THRESHOLD`; eviction is implicit on `create`, no daemon, no `enroot gc` command). Default backend (`dir`) is unchanged. -- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plan A is merged** on `zenroot/main` (PR [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1)); B–F are still design-only. +- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, and F are merged** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3)); B, C, D are still design-only. ## Conventions diff --git a/doc/plans/2026-04-29-zfs-f-docker-load.md b/doc/plans/2026-04-29-zfs-f-docker-load.md index d5803d4..0c9e45a 100644 --- a/doc/plans/2026-04-29-zfs-f-docker-load.md +++ b/doc/plans/2026-04-29-zfs-f-docker-load.md @@ -2,26 +2,33 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** When `ENROOT_STORAGE_BACKEND=zfs`, `enroot load docker://...` stops requiring `ENROOT_NATIVE_OVERLAYFS=y`. Layers are stacked as a chain of ZFS clones (mirroring Docker's own `zfs` storage driver): each layer is `zfs clone parent@done`, the layer tarball is extracted into the clone with whiteout handling, then `zfs snapshot @done`. The leaf snapshot becomes the cached template's `@pristine`, and a clone of that becomes the user's container. +**Goal:** When `ENROOT_STORAGE_BACKEND=zfs`, `enroot load docker://...` no longer requires `ENROOT_NATIVE_OVERLAYFS=y`. The merged image is materialized into a ZFS template dataset (cached by image config digest) and the user's container is a `zfs clone` of it. Default `dir` backend behavior is preserved byte-for-byte and still requires `ENROOT_NATIVE_OVERLAYFS=y`. -**Architecture:** Add `zfs::stack_layers` that takes the same layer-tarball list `docker::_prepare_layers` produces and replays it onto a fresh chain of ZFS datasets. Modify `docker::load` to dispatch on backend: existing `enroot-mksquashovlfs` path stays for the `dir` backend; ZFS path uses `zfs::stack_layers` and skips the `ENROOT_NATIVE_OVERLAYFS=y` precondition. +**Architecture:** `docker::_prepare_layers` already does the heavy lifting — for each layer it `mkdir N`, untars the layer's tarball into directory `N/`, and runs `enroot-aufs2ovlfs N` to convert AUFS-style whiteouts to overlayfs whiteouts. After it returns, the cwd contains directories `0/` (synthetic config layer with `/etc/{rc,fstab,environment}` from `docker::configure`) and `1/` … `N/` (extracted, whiteout-converted layer trees). The dir-backend `docker::load` then runs `enroot-nsenter --user --remap-root` + `mount -t overlay lowerdir=0:1:…:N rootfs` and tar-pipes the merged view into `${name}/`. + +The ZFS path reuses that exact merge logic — the only thing that changes is the *destination* of the tar-pipe: instead of writing into a regular directory under `${ENROOT_DATA_PATH}`, we write into the mountpoint of a freshly-created ZFS template dataset, then snapshot `@pristine` and clone for the user. The clone (and its mountpoint resolution) happens **outside** the user namespace because zfs(8) cannot enumerate datasets from inside one (see Plan E for the full background). + +To keep `docker.sh` minimally invasive, all ZFS-specific lifecycle logic lives in `src/storage_zfs.sh` behind two helpers; `docker::load` only dispatches. **Depends on:** Plan A. -**Prerequisite host setup:** Same as Plan A. Test image: `alpine` from Docker Hub (small, exercises whiteouts via standard Docker tooling). +**Prerequisite host setup:** Same as Plan A. Test images: `docker://alpine` (single layer, fast), `docker://debian:stable-slim` (multi-layer, exercises whiteouts). --- ## Files -- **Modify:** `src/storage_zfs.sh` — add `zfs::stack_layers`, `zfs::extract_layer_tarball`. -- **Modify:** `src/docker.sh:488-548` (`docker::load`) — backend dispatch. +- **Modify:** `src/storage_zfs.sh` — add `zfs::container_check` (early existence-or-destroy gate) and `zfs::docker_install_from_layers` (template lifecycle + overlay merge + clone-for-user). +- **Modify:** `src/docker.sh` (`docker::load`) — relax the `ENROOT_NATIVE_OVERLAYFS=y` precondition when ZFS is enabled; replace the inline existence-check and merge blocks with two-line backend dispatches calling the new helpers. +- **Modify:** `doc/zfs.md` and `CLAUDE.md` — status notes. + +`docker::_prepare_layers`, `docker::configure`, and the existing dir-backend overlay path are **not** modified. --- -### Task 1: Add `zfs::extract_layer_tarball` (whiteout-aware) +### Task 1: Add `zfs::container_check` -Docker layer tarballs use AUFS-style whiteouts: `.wh.foo` means "delete `foo` from lower layers"; `.wh..wh..opq` in a directory means "ignore everything from lower layers in this directory." When stacking with overlayfs, the kernel handles these. With ZFS clones, we replicate the semantics manually. +A small early-exit gate that errors (or destroys with `--force`) if a container of the given name already exists in the ZFS store. Used so `docker::load` can fail fast before downloading layers it would discard. **Files:** - Modify: `src/storage_zfs.sh` (append) @@ -31,172 +38,88 @@ Docker layer tarballs use AUFS-style whiteouts: `.wh.foo` means "delete `foo` fr Append to `src/storage_zfs.sh`: ```bash -# Extracts a single Docker layer tarball into the given mountpoint with -# whiteout/opaque-marker handling, suitable for use on top of a parent layer's -# ZFS clone. Handles both compressed and uncompressed tarballs. -zfs::extract_layer_tarball() { - local -r tarball="$1" mountpoint="$2" - - common::checkcmd tar awk find - - # Pass 1: extract everything except whiteout markers. - tar --numeric-owner -C "${mountpoint}" -xpf "${tarball}" \ - --exclude='.wh.*' --exclude='.wh..wh..opq' 2> /dev/null || \ - tar --numeric-owner -C "${mountpoint}" -xpf "${tarball}" \ - --exclude='.wh.*' --exclude='.wh..wh..opq' # surface real errors on retry - - # Pass 2: list whiteout markers and apply them to the tree. - tar -tf "${tarball}" 2> /dev/null | awk ' - /\/\.wh\..*$/ { sub(/\.wh\.([^/]+)$/, "\\1"); print "del\t"$0; next } - /^\.wh\..*$/ { sub(/^\.wh\./, ""); print "del\t"$0; next } - /\.wh\..wh..opq$/ { sub(/\.wh\..wh..opq$/, ""); print "opq\t"$0; next } - ' | while IFS=$'\t' read -r kind path; do - case "${kind}" in - del) - rm -rf "${mountpoint}/${path}" 2> /dev/null || : - ;; - opq) - # Opaque dir: clear everything in this directory from lower layers. - # The directory itself was already created by this layer's pass 1 - # (or already exists from a parent). Remove all children, then let - # pass 1's content (which we've already laid down) repopulate. - find "${mountpoint}/${path}" -mindepth 1 -maxdepth 1 -exec rm -rf {} + 2> /dev/null || : - # Re-extract the directory's contents from this tar without the wh markers. - tar --numeric-owner -C "${mountpoint}" -xpf "${tarball}" \ - --exclude='.wh.*' --exclude='.wh..wh..opq' \ - "${path}" 2> /dev/null || : - ;; - esac - done +# Errors (or destroys with --force) if a container with this name already exists +# in the ZFS store. Used as an early-exit gate before doing expensive work +# (e.g. downloading Docker layers we'd just throw away). +zfs::container_check() { + local -r name="$1" + local target + target="$(zfs::store_dataset)/${name}" + if zfs list -H "${target}" > /dev/null 2>&1; then + if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then + common::err "Container already exists: ${name}" + fi + zfs destroy -r "${target}" + fi } ``` -- [ ] **Step 1.2: Verify with a hand-built tarball** - -```sh -mkdir /tmp/wh-test && cd /tmp/wh-test -mkdir -p layer1/usr/bin layer2 -echo "old" > layer1/usr/bin/foo -echo "x" > layer1/keep_me -( cd layer2 && touch usr/bin/.wh.foo ) # delete /usr/bin/foo -tar --numeric-owner -C layer1 -cf l1.tar . -tar --numeric-owner -C layer2 -cf l2.tar . - -mkdir target -tar -C target -xpf l1.tar -ls target/usr/bin/foo # exists - -bash -c 'source ${ENROOT_LIBRARY_PATH}/common.sh - source ${ENROOT_LIBRARY_PATH}/storage_zfs.sh - zfs::extract_layer_tarball /tmp/wh-test/l2.tar /tmp/wh-test/target' -ls /tmp/wh-test/target/usr/bin/foo 2>&1 | grep -q "No such" && echo "deleted OK" -ls /tmp/wh-test/target/keep_me && echo "preserved OK" -cd && rm -rf /tmp/wh-test -``` - -Expected: `deleted OK` and `preserved OK`. - -- [ ] **Step 1.3: Commit** +- [ ] **Step 1.2: Commit** ```sh git add src/storage_zfs.sh -git commit -s -m "Add zfs::extract_layer_tarball with whiteout/opaque handling" +git commit -s -m "Add zfs::container_check early-exit helper" ``` --- -### Task 2: Add `zfs::stack_layers` +### Task 2: Add `zfs::docker_install_from_layers` + +The full ZFS template-fill-and-clone lifecycle for the Docker case. Designed to be called from `docker::load` *after* `docker::_prepare_layers` has populated the cwd with directories `0/`, `1/`, …, `N/`. **Files:** - Modify: `src/storage_zfs.sh` (append) -- [ ] **Step 2.1: Add the function** +- [ ] **Step 2.1: Add the helper** Append to `src/storage_zfs.sh`: ```bash -# Stacks an ordered list of Docker layer tarballs as a chain of ZFS datasets. -# Returns the final template's name (an existing template will be reused if its -# leaf snapshot already exists). +# Materializes the merged Docker rootfs into a ZFS template (cached by +# cache_key) and clones it as the user's named container. Designed to be +# called from docker::load AFTER docker::_prepare_layers has populated the +# cwd with extracted, whiteout-converted layer directories 0/, 1/, ..., N/. # # Inputs: -# $1 - cache key (typically sha256 of the manifest digest list, computed by caller) -# $2 - tab-separated list of layer tarball paths (one per line, in stack order -# from base to top), passed via stdin +# $1 cache_key - sha256 of the image config blob (a stable per-image key) +# $2 layer_count - the N from _prepare_layers (count of layer directories) +# $3 unpriv - "y" or "" — whether to enter a new user namespace +# $4 name - the user-visible container name (no slashes) # -# Output (stdout): the template dataset name. -zfs::stack_layers() { - local -r cache_key="$1" - local -r store=$(zfs::store_dataset) - local -r template="${store}/${zfs_template_subdir}/${cache_key}" - local -r tmp="${template}.tmp" - local -r snap="${template}@${zfs_pristine_snap}" - local i timeout=600 - local layers parent layer mountpoint i_layer=0 - - zfs::sweep_templates 2> /dev/null || : +# Atomicity: races on the same cache_key are resolved via a per-key .tmp +# dataset lock; losers wait for @pristine. ENOSPC mid-merge destroys the +# .tmp so a retry can run. +zfs::docker_install_from_layers() { + local -r cache_key="$1" layer_count="$2" unpriv="$3" name="$4" + local store template tmp snap mountpoint i=0 + store=$(zfs::store_dataset) + template="${store}/${zfs_template_subdir}/${cache_key}" + tmp="${template}.tmp" + snap="${template}@${zfs_pristine_snap}" if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then - zfs::touch_template "${template}" 2> /dev/null || : - printf "%s" "${template}" - return - fi - - # Read layer paths from stdin. - readarray -t layers - - if zfs create -p "${tmp}" 2> /dev/null; then - parent="${tmp}" - for layer in "${layers[@]}"; do - [ -z "${layer}" ] && continue - if [ "${i_layer}" -eq 0 ]; then - # First layer extracts directly into the .tmp dataset. - mountpoint=$(zfs get -H -o value mountpoint "${parent}") - common::log INFO "Stacking layer 1/${#layers[@]}..." - zfs::extract_layer_tarball "${layer}" "${mountpoint}" - zfs snapshot "${parent}@layer-${i_layer}" - else - # Subsequent layers clone the previous snapshot, then extract. - local child="${tmp}-l${i_layer}" - zfs clone "${parent}@layer-$((i_layer - 1))" "${child}" - mountpoint=$(zfs get -H -o value mountpoint "${child}") - common::log INFO "Stacking layer $((i_layer + 1))/${#layers[@]}..." - zfs::extract_layer_tarball "${layer}" "${mountpoint}" - zfs snapshot "${child}@layer-${i_layer}" - parent="${child}" - fi - i_layer=$((i_layer + 1)) - done - - # The 'parent' variable now points at the leaf clone. Promote it so it - # becomes the new template root, then clean up the intermediate chain. - if [ "${parent}" != "${tmp}" ]; then - zfs promote "${parent}" + common::log INFO "Reusing cached template ${cache_key:0:12}" + elif zfs create -p "${tmp}" 2> /dev/null; then + mountpoint=$(zfs get -H -o value mountpoint "${tmp}") + mkdir -p rootfs + if ! enroot-nsenter ${unpriv:+--user} --mount --remap-root \ + bash -c "mount --make-rprivate / && mount -t overlay overlay -o lowerdir=0:$(seq -s: 1 "${layer_count}") rootfs && + tar --numeric-owner -C rootfs/ --mode=u-s,g-s -cpf - . | tar --numeric-owner -C '${mountpoint}/' -xpf -"; then + zfs destroy -r "${tmp}" 2> /dev/null || : + common::err "Failed to merge Docker layers into ZFS template" fi - zfs rename "${parent}" "${template}" + zfs rename "${tmp}" "${template}" zfs snapshot "${snap}" - - # Best-effort cleanup of intermediate datasets/snapshots. - local left - for left in $(zfs list -H -o name -r "${store}/${zfs_template_subdir}" | grep -E "^${tmp//./\\.}(-l[0-9]+)?\$"); do - zfs destroy -r "${left}" 2> /dev/null || : - done - zfs set readonly=on "${template}" - zfs::touch_template "${template}" 2> /dev/null || : - printf "%s" "${template}" - return + else + # Lost the race or stale .tmp — wait for @pristine. + while ! zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; do + sleep 1 + ((i++ < 600)) || common::err "Timed out waiting for Docker template: ${template}" + done fi - # Lost the race or stale .tmp — wait for @pristine. - for ((i = 0; i < timeout; i++)); do - if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then - printf "%s" "${template}" - return - fi - sleep 1 - done - common::err "Timed out waiting for layer-stack template: ${template}" + zfs::clone_container "${template}" "${name}" } ``` @@ -204,51 +127,71 @@ zfs::stack_layers() { ```sh git add src/storage_zfs.sh -git commit -s -m "Add zfs::stack_layers for Docker layer-stack templates" +git commit -s -m "Add zfs::docker_install_from_layers helper" ``` --- -### Task 3: Branch `docker::load` on backend +### Task 3: Wire the helpers into `docker::load` + +The dir-backend keeps its existing inline path; the ZFS branch is two dispatch calls. **Files:** -- Modify: `src/docker.sh:488-548` +- Modify: `src/docker.sh` (`docker::load`) -- [ ] **Step 3.1: Replace the `ENROOT_NATIVE_OVERLAYFS=y` precondition with a backend-conditional check** +- [ ] **Step 3.1: Relax the `ENROOT_NATIVE_OVERLAYFS=y` precondition** -In `src/docker.sh:488-495`, change: +In `src/docker.sh`, change the early precondition check (around line 493): ```bash -docker::load() ( - local -r uri="$1" - local name="$2" arch="$3" - local user= registry= image= tag= tmpdir= config= layer_count= - if [ -z "${ENROOT_NATIVE_OVERLAYFS-}" ]; then common::err "ENROOT_NATIVE_OVERLAYFS=y is required for enroot load" fi - ... ``` to: ```bash -docker::load() ( - local -r uri="$1" - local name="$2" arch="$3" - local user= registry= image= tag= tmpdir= config= layer_count= - if ! zfs::enabled && [ -z "${ENROOT_NATIVE_OVERLAYFS-}" ]; then common::err "ENROOT_NATIVE_OVERLAYFS=y or ENROOT_STORAGE_BACKEND=zfs is required for enroot load" fi - ... ``` -- [ ] **Step 3.2: Add backend dispatch for the layer-stacking step** +- [ ] **Step 3.2: Dispatch the existence check on backend** + +Replace the existing-rootfs check (around lines 517–524): + +```bash + name=$(common::realpath "${ENROOT_DATA_PATH}/${name}") + if [ -e "${name}" ]; then + if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then + common::err "File already exists: ${name}" + else + common::rmall "${name}" + fi + fi +``` + +with: + +```bash + if zfs::enabled; then + zfs::container_check "${name}" + else + name=$(common::realpath "${ENROOT_DATA_PATH}/${name}") + if [ -e "${name}" ]; then + if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then + common::err "File already exists: ${name}" + else + common::rmall "${name}" + fi + fi + fi +``` -In `src/docker.sh`, after `docker::_prepare_layers` returns and before the existing `enroot-nsenter ... overlay` block (around line 545), add a backend dispatch. +- [ ] **Step 3.3: Dispatch the merge step on backend** -Locate the existing block (lines 535-547): +Replace the existing merge block (around lines 535–547) — the existing dir-backend `mkdir -p rootfs ... enroot-nsenter ... mount -t overlay ... tar pipe` — with: ```bash # Create the final filesystem by overlaying all the layers and copying to target rootfs. @@ -259,149 +202,139 @@ Locate the existing block (lines 535-547): unpriv=y fi - # Create a mount namespace and overlay mount - mkdir -p rootfs "${name}" - enroot-nsenter ${unpriv:+--user} --mount --remap-root \ - bash -c "mount --make-rprivate / && mount -t overlay overlay -o lowerdir=0:$(seq -s: 1 "${layer_count}") rootfs && - tar --numeric-owner -C rootfs/ --mode=u-s,g-s -cpf - . | tar --numeric-owner -C '${name}/' -xpf -" -) -``` - -Wrap it in a backend conditional: - -```bash - common::log INFO "Loading container root filesystem..." NL - if zfs::enabled; then - # ZFS path: stack layers as a chain of clones; final template, then clone for the user. - local cache_key template - # The cache key is the sha256 of the layer-tarball list (stable per-image). - cache_key=$(printf "%s\n" 0 $(seq 1 "${layer_count}") | xargs -I{} sha256sum {} 2>/dev/null \ - | awk '{print $1}' | sha256sum | awk '{print $1}') - - # Build the absolute paths to the layer tarballs already prepared in $tmpdir. - # docker::_prepare_layers leaves layers at "0", "1", ..., "${layer_count}" relative - # to the cwd at this point. Pass them in stack order (0 first if present, then 1..N). - local -a layer_paths=() - [ -e 0 ] && layer_paths+=("$(common::realpath 0)") - for i in $(seq 1 "${layer_count}"); do - [ -e "${i}" ] && layer_paths+=("$(common::realpath "${i}")") - done - - template=$(printf "%s\n" "${layer_paths[@]}" | zfs::stack_layers "${cache_key}") - zfs::clone_container "${template}" "${name##*/}" + zfs::docker_install_from_layers "${config}" "${layer_count}" "${unpriv}" "${name}" else - # Check if we're running unprivileged. - if [ "${EUID}" -ne 0 ]; then - unpriv=y - fi - # Create a mount namespace and overlay mount mkdir -p rootfs "${name}" enroot-nsenter ${unpriv:+--user} --mount --remap-root \ bash -c "mount --make-rprivate / && mount -t overlay overlay -o lowerdir=0:$(seq -s: 1 "${layer_count}") rootfs && tar --numeric-owner -C rootfs/ --mode=u-s,g-s -cpf - . | tar --numeric-owner -C '${name}/' -xpf -" fi -) ``` -NOTE: `docker::_prepare_layers` writes layer tarballs into the cwd (`$tmpdir`) under integer names; the existing overlay path already references them as `0:$(seq -s: 1 "${layer_count}")`. The exact naming convention (`0`, `1`, …, `N`, with `0` being the empty/lower marker) should be confirmed by reading `docker::_prepare_layers` (`src/docker.sh:306`) before implementing this task. If the naming differs, adjust the layer-paths construction. +- [ ] **Step 3.4: Add `unpriv=` to the function's locals** -- [ ] **Step 3.3: Verify ZFS-backed `enroot load`** +Near the top of `docker::load`'s local declarations, add `unpriv=` so the variable is initialized before the `if [ "${EUID}" -ne 0 ]` block: -```sh -export ENROOT_STORAGE_BACKEND=zfs -export ENROOT_DATA_PATH=/srv/enroot/$USER -unset ENROOT_NATIVE_OVERLAYFS -/tmp/enroot/usr/bin/enroot load docker://alpine -n alpine_loaded -ls /srv/enroot/$USER/alpine_loaded/etc/os-release -zfs list | grep alpine_loaded -/tmp/enroot/usr/bin/enroot remove -f alpine_loaded +```bash + local user= registry= image= tag= tmpdir= config= layer_count= unpriv= ``` -Expected: load succeeds without `ENROOT_NATIVE_OVERLAYFS=y`; clone is listed; os-release readable. - -- [ ] **Step 3.4: Verify `dir` backend `enroot load` still requires `ENROOT_NATIVE_OVERLAYFS=y` (no regression)** +- [ ] **Step 3.5: Commit** ```sh -unset ENROOT_STORAGE_BACKEND -unset ENROOT_NATIVE_OVERLAYFS -/tmp/enroot/usr/bin/enroot load docker://alpine -n a 2>&1 | grep -q "is required" && echo OK -ENROOT_NATIVE_OVERLAYFS=y /tmp/enroot/usr/bin/enroot load docker://alpine -n a -/tmp/enroot/usr/bin/enroot remove -f a +git add src/docker.sh +git commit -s -m "Branch docker::load on ENROOT_STORAGE_BACKEND via storage_zfs.sh helpers" ``` -Expected: first invocation errors with the precondition message (`OK`); second succeeds. +--- -- [ ] **Step 3.5: Verify whiteouts work end-to-end** +### Task 4: Verify single-layer image (alpine) -Use a test image with known whiteouts (any multi-layer image where a later layer deletes a file from an earlier one — `python:3-slim` is a common example): +- [ ] **Step 4.1: ZFS load without `ENROOT_NATIVE_OVERLAYFS=y`** ```sh -export ENROOT_STORAGE_BACKEND=zfs -/tmp/enroot/usr/bin/enroot load docker://python:3-slim -n py -ls /srv/enroot/$USER/py/usr/bin/python3 && echo "ok" -# Check no .wh. files leaked through: -find /srv/enroot/$USER/py -name '.wh.*' | head && echo "BAD" || echo "clean" -/tmp/enroot/usr/bin/enroot remove -f py +sudo PATH=/tmp/enroot/usr/bin:$PATH ENROOT_STORAGE_BACKEND=zfs ENROOT_DATA_PATH=/srv/enroot/$USER ENROOT_NATIVE_OVERLAYFS=no \ + /tmp/enroot/usr/bin/enroot load -n alpine_loaded docker://alpine +ls /srv/enroot/$USER/alpine_loaded/etc/os-release +sudo PATH=/tmp/enroot/usr/bin:$PATH ENROOT_STORAGE_BACKEND=zfs ENROOT_DATA_PATH=/srv/enroot/$USER \ + /tmp/enroot/usr/bin/enroot start alpine_loaded /bin/cat /etc/os-release | head -2 +sudo PATH=/tmp/enroot/usr/bin:$PATH ENROOT_STORAGE_BACKEND=zfs ENROOT_DATA_PATH=/srv/enroot/$USER \ + /tmp/enroot/usr/bin/enroot remove -f alpine_loaded ``` -Expected: `ok` and `clean`. +Expected: load succeeds; `os-release` readable; `start` prints alpine os-release; `remove` succeeds. + +--- -- [ ] **Step 3.6: Commit** +### Task 5: Verify multi-layer image with whiteouts (debian:stable-slim) + +- [ ] **Step 5.1: Multi-layer load** ```sh -git add src/docker.sh -git commit -s -m "Branch docker::load on ENROOT_STORAGE_BACKEND; lift overlay precondition for ZFS" +sudo PATH=/tmp/enroot/usr/bin:$PATH ENROOT_STORAGE_BACKEND=zfs ENROOT_DATA_PATH=/srv/enroot/$USER ENROOT_NATIVE_OVERLAYFS=no \ + /tmp/enroot/usr/bin/enroot load -n debian_loaded docker://debian:stable-slim +ls /srv/enroot/$USER/debian_loaded/usr/bin/dpkg && echo "dpkg present" +count=$(sudo find /srv/enroot/$USER/debian_loaded -name '.wh.*' 2>/dev/null | wc -l) +[ "${count}" = "0" ] && echo "no aufs whiteouts OK" +sudo PATH=/tmp/enroot/usr/bin:$PATH ENROOT_STORAGE_BACKEND=zfs ENROOT_DATA_PATH=/srv/enroot/$USER \ + /tmp/enroot/usr/bin/enroot start debian_loaded /bin/cat /etc/os-release | grep PRETTY +sudo PATH=/tmp/enroot/usr/bin:$PATH ENROOT_STORAGE_BACKEND=zfs ENROOT_DATA_PATH=/srv/enroot/$USER \ + /tmp/enroot/usr/bin/enroot remove -f debian_loaded ``` +Expected: load succeeds; `dpkg present`; `no aufs whiteouts OK`; `start` prints Debian os-release. + --- -### Task 4: Document Plan F as implemented +### Task 6: Verify cache reuse and `dir`-backend regression -**Files:** -- Modify: `doc/zfs.md` +- [ ] **Step 6.1: Cache reuse on second load** -- [ ] **Step 4.1: Update status note** +```sh +sudo PATH=/tmp/enroot/usr/bin:$PATH ENROOT_STORAGE_BACKEND=zfs ENROOT_DATA_PATH=/srv/enroot/$USER ENROOT_NATIVE_OVERLAYFS=no \ + /tmp/enroot/usr/bin/enroot load -n a docker://alpine +sudo PATH=/tmp/enroot/usr/bin:$PATH ENROOT_STORAGE_BACKEND=zfs ENROOT_DATA_PATH=/srv/enroot/$USER ENROOT_NATIVE_OVERLAYFS=no \ + bash -c 'time /tmp/enroot/usr/bin/enroot load -n b docker://alpine' 2>&1 | tail -5 +sudo PATH=/tmp/enroot/usr/bin:$PATH ENROOT_STORAGE_BACKEND=zfs ENROOT_DATA_PATH=/srv/enroot/$USER \ + /tmp/enroot/usr/bin/enroot remove -f a b +``` -Update `doc/zfs.md` to mark Plan F (Docker layer stacking on ZFS) as landed. +Expected: second load logs `Reusing cached template ...` and completes in well under a second. -- [ ] **Step 4.2: End-to-end smoke** +- [ ] **Step 6.2: Dir-backend without `ENROOT_NATIVE_OVERLAYFS=y` errors** ```sh -export ENROOT_STORAGE_BACKEND=zfs -unset ENROOT_NATIVE_OVERLAYFS -/tmp/enroot/usr/bin/enroot load docker://alpine -n a -/tmp/enroot/usr/bin/enroot start a /bin/cat /etc/os-release -/tmp/enroot/usr/bin/enroot load docker://alpine -n b # second time should reuse template -zfs list -t all | grep templates | wc -l # should be 1, not 2 -/tmp/enroot/usr/bin/enroot remove -f a b +sudo PATH=/tmp/enroot/usr/bin:$PATH ENROOT_DATA_PATH=$HOME/.local/share/enroot ENROOT_NATIVE_OVERLAYFS=no \ + /tmp/enroot/usr/bin/enroot load -n a docker://alpine 2>&1 | grep -q 'is required' && echo OK ``` -Expected: both loads succeed; only one template remains. +Expected: `OK` — precondition error fires. + +- [ ] **Step 6.3: Dir-backend with `ENROOT_NATIVE_OVERLAYFS=y` works (regression)** + +```sh +sudo PATH=/tmp/enroot/usr/bin:$PATH ENROOT_DATA_PATH=$HOME/.local/share/enroot ENROOT_NATIVE_OVERLAYFS=y \ + /tmp/enroot/usr/bin/enroot load -n alpine_dir docker://alpine +ls $HOME/.local/share/enroot/alpine_dir/etc/os-release && echo "dir-backend OK" +sudo PATH=/tmp/enroot/usr/bin:$PATH ENROOT_DATA_PATH=$HOME/.local/share/enroot \ + /tmp/enroot/usr/bin/enroot remove -f alpine_dir +``` + +Expected: load succeeds; rootfs readable. + +--- + +### Task 7: Document Plan F as implemented and open PR + +**Files:** +- Modify: `doc/zfs.md` — flip status note to "Plans A, E, F implemented"; rewrite the `enroot load docker://` row of the "Where the ZFS backend is used" table. +- Modify: `CLAUDE.md` — update the "Active design proposals" line. -- [ ] **Step 4.3: Commit** +- [ ] **Step 7.1: Commit** ```sh -git add doc/zfs.md +git add doc/zfs.md CLAUDE.md git commit -s -m "Mark Plan F (Docker load ZFS path) as implemented" +git push -u origin feature/zfs-f-docker-load +gh pr create --repo zeroae/enroot --base zenroot/main --head feature/zfs-f-docker-load \ + --title "Plan F: ZFS path for enroot load docker://" --body "..." ``` --- ## Self-review checklist -- [x] Spec coverage: ZFS layer-stacking instead of mksquashovlfs (T2, T3.2); `ENROOT_NATIVE_OVERLAYFS=y` precondition lifted on ZFS (T3.1, T3.3); whiteout & opaque-dir handling (T1, T3.5); cache reuse across loads of same image (T2 fast-path, T4.2 verifies); `dir` backend behavior unchanged (T3.4 regression check). -- [x] Type consistency: `zfs::extract_layer_tarball`, `zfs::stack_layers` defined in T1, T2; both used in T3. +- [x] Spec coverage: precondition lifted on ZFS (T3.1), early-exit existence check (T1, T3.2), single-pass overlay merge into ZFS clone (T2, T3.3), cache reuse (T6.1), whiteouts handled correctly (T5), dir regression both directions (T6.2, T6.3). +- [x] All ZFS-specific lifecycle logic lives in `src/storage_zfs.sh`; `docker.sh` only dispatches. Helpers used (`zfs::container_check`, `zfs::docker_install_from_layers`) defined in T1, T2 and called in T3. - [x] No placeholders. -## Known limitations & open questions +## Known limitations -- **Step 3.2 has an explicit caveat** about `docker::_prepare_layers`'s on-disk naming. The implementer must read `src/docker.sh:306-330` and confirm before writing the layer-paths array. If `_prepare_layers` writes to a different layout, the array construction must be adjusted accordingly. -- **`zfs promote` on the leaf** is the canonical way to "flatten" a clone chain into a standalone dataset. We do this so the intermediate `-l1`, `-l2`, … datasets can be destroyed and the cache stores only the final template. If `zfs promote` is unavailable for the user's delegations, an alternative is to keep the chain alive and accept extra dataset objects (no functional impact, just `zfs list` clutter). -- **Per-layer xattr handling** (capability bits, immutable flags, security.* attrs) is the same as Docker's own `zfs` driver — `tar --numeric-owner` plus the `xattr=sa` filesystem property carries them. Note in `doc/zfs.md` admin recipe (already covered there as a default). -- **Concurrent `enroot load` of the same image** is race-safe via the same `.tmp` lock as Plan A's `ensure_template` — losers wait for `@pristine`. -- **Sparse files in layers** are not specially handled; tar's default sparse handling applies. Likely fine for v1. +- **No per-layer dedup across distinct images.** Each distinct image gets its own template at the merged-rootfs level. ZFS `dedup=on` on the templates dataset (admin opt-in) recovers most of this savings via block-level dedup; explicit per-layer-dataset chaining was rejected here in favor of staying close to the existing `_prepare_layers` flow. +- **The merge runs inside `enroot-nsenter --user --remap-root`**, same as the dir backend. The kernel's overlay support (or `fuse-overlayfs` if `ENROOT_NATIVE_OVERLAYFS` is unset) is still the merge engine; we just redirect the tar-pipe target. +- **Concurrent loads of the same image** are race-safe via the same `.tmp` lock pattern as Plan A's `ensure_template`: losers wait for `@pristine`. ## Execution Handoff diff --git a/doc/zfs.md b/doc/zfs.md index 44a6c87..c0a2159 100644 --- a/doc/zfs.md +++ b/doc/zfs.md @@ -1,6 +1,6 @@ # ZFS storage backend -This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation) and E (ephemeral start) are implemented**: `enroot create`, `enroot remove`, and ephemeral `enroot start ` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`. The remaining substitutions (template warm/cold lifecycle, `.zfs` file format, `zfs://` URI, Docker layer stacking on ZFS) are tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS. +This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start `, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`. The remaining substitutions (template warm/cold lifecycle, `.zfs` file format, `zfs://` URI) are tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS. ## Motivation @@ -222,7 +222,7 @@ When `ENROOT_STORAGE_BACKEND=zfs`, ZFS substitutes for the existing storage code | ------ | ------ | ------ | | `enroot create foo.sqsh` | `unsquashfs` into `/` | Extract once into `templates/`, `zfs clone templates/@pristine /`. | | `enroot start foo.sqsh` (ephemeral; no prior `create`) | `squashfuse` lower layer + overlay upper layer, with kernel `overlay` or `fuse-overlayfs` selected by `ENROOT_NATIVE_OVERLAYFS`. | Ensure template, `zfs clone @pristine` to a unique ephemeral name, `zfs destroy` on exit. | -| `enroot load docker://` (fetch + create in one step) | Layer-stack via `enroot-mksquashovlfs` overlay; requires `ENROOT_NATIVE_OVERLAYFS=y`. | Stack Docker layers as a chain of ZFS datasets — for each layer in order, `zfs clone parent@done` produces a writable child, the layer tarball is extracted into it with whiteout handling, then `zfs snapshot @done`. The leaf snapshot becomes the template's `@pristine`. Mirrors Docker's own `zfs` storage driver. | +| `enroot load docker://` (fetch + create in one step) | Layer-stack via `enroot-mksquashovlfs` overlay; requires `ENROOT_NATIVE_OVERLAYFS=y`. | `docker::_prepare_layers` produces extracted, whiteout-converted layer directories `0/`, `1/`, …, `N/` exactly as on the `dir` backend. The merge step reuses the same `enroot-nsenter --user --remap-root` + `mount -t overlay lowerdir=0:1:…:N` pipeline, but the tar-pipe is redirected from a regular directory into the mountpoint of a freshly-created template clone (keyed by image config sha). Cache hits skip the merge entirely. `ENROOT_NATIVE_OVERLAYFS=y` is **not** required; ZFS replaces the precondition. | `ENROOT_NATIVE_OVERLAYFS` keeps its current meaning when the backend is `dir`. When the backend is `zfs`, the knob is irrelevant for the three substituted paths above (overlay is not used), but it continues to control overlay choice for any code path that does not yet have a ZFS substitution. diff --git a/src/docker.sh b/src/docker.sh index f67e61d..06fae9b 100644 --- a/src/docker.sh +++ b/src/docker.sh @@ -488,10 +488,10 @@ docker::import() ( docker::load() ( local -r uri="$1" local name="$2" arch="$3" - local user= registry= image= tag= tmpdir= config= layer_count= + local user= registry= image= tag= tmpdir= config= layer_count= unpriv= - if [ -z "${ENROOT_NATIVE_OVERLAYFS-}" ]; then - common::err "ENROOT_NATIVE_OVERLAYFS=y is required for enroot load" + if ! zfs::enabled && [ -z "${ENROOT_NATIVE_OVERLAYFS-}" ]; then + common::err "ENROOT_NATIVE_OVERLAYFS=y or ENROOT_STORAGE_BACKEND=zfs is required for enroot load" fi common::checkcmd curl grep awk jq parallel tar "${ENROOT_GZIP_PROGRAM}" find zstd @@ -514,12 +514,16 @@ docker::load() ( name="${display_image////+}${tag:++${tag}}" fi - name=$(common::realpath "${ENROOT_DATA_PATH}/${name}") - if [ -e "${name}" ]; then - if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then - common::err "File already exists: ${name}" - else - common::rmall "${name}" + if zfs::enabled; then + zfs::container_check "${name}" + else + name=$(common::realpath "${ENROOT_DATA_PATH}/${name}") + if [ -e "${name}" ]; then + if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then + common::err "File already exists: ${name}" + else + common::rmall "${name}" + fi fi fi @@ -540,11 +544,15 @@ docker::load() ( unpriv=y fi - # Create a mount namespace and overlay mount - mkdir -p rootfs "${name}" - enroot-nsenter ${unpriv:+--user} --mount --remap-root \ - bash -c "mount --make-rprivate / && mount -t overlay overlay -o lowerdir=0:$(seq -s: 1 "${layer_count}") rootfs && - tar --numeric-owner -C rootfs/ --mode=u-s,g-s -cpf - . | tar --numeric-owner -C '${name}/' -xpf -" + if zfs::enabled; then + zfs::docker_install_from_layers "${config}" "${layer_count}" "${unpriv}" "${name}" + else + # Create a mount namespace and overlay mount + mkdir -p rootfs "${name}" + enroot-nsenter ${unpriv:+--user} --mount --remap-root \ + bash -c "mount --make-rprivate / && mount -t overlay overlay -o lowerdir=0:$(seq -s: 1 "${layer_count}") rootfs && + tar --numeric-owner -C rootfs/ --mode=u-s,g-s -cpf - . | tar --numeric-owner -C '${name}/' -xpf -" + fi ) docker::daemon::import() ( diff --git a/src/storage_zfs.sh b/src/storage_zfs.sh index b6e06ae..5926172 100644 --- a/src/storage_zfs.sh +++ b/src/storage_zfs.sh @@ -163,3 +163,65 @@ zfs::ephemeral_destroy() { [ -z "${clone}" ] && return zfs destroy "${clone}" 2> /dev/null || : } + +# Errors (or destroys with --force) if a container with this name already exists +# in the ZFS store. Used as an early-exit gate before doing expensive work +# (e.g. downloading Docker layers we'd just throw away). +zfs::container_check() { + local -r name="$1" + local target + target="$(zfs::store_dataset)/${name}" + if zfs list -H "${target}" > /dev/null 2>&1; then + if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then + common::err "Container already exists: ${name}" + fi + zfs destroy -r "${target}" + fi +} + +# Materializes the merged Docker rootfs into a ZFS template (cached by +# cache_key) and clones it as the user's named container. Designed to be +# called from docker::load AFTER docker::_prepare_layers has populated the +# cwd with extracted, whiteout-converted layer directories 0/, 1/, ..., N/. +# +# Inputs: +# $1 cache_key - sha256 of the image config blob (a stable per-image key) +# $2 layer_count - the N from _prepare_layers (count of layer directories) +# $3 unpriv - "y" or "" — whether to enter a new user namespace +# $4 name - the user-visible container name (no slashes) +# +# Atomicity: races on the same cache_key are resolved via a per-key .tmp +# dataset lock; losers wait for @pristine. ENOSPC mid-merge destroys the +# .tmp so a retry can run. +zfs::docker_install_from_layers() { + local -r cache_key="$1" layer_count="$2" unpriv="$3" name="$4" + local store template tmp snap mountpoint i=0 + store=$(zfs::store_dataset) + template="${store}/${zfs_template_subdir}/${cache_key}" + tmp="${template}.tmp" + snap="${template}@${zfs_pristine_snap}" + + if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then + common::log INFO "Reusing cached template ${cache_key:0:12}" + elif zfs create -p "${tmp}" 2> /dev/null; then + mountpoint=$(zfs get -H -o value mountpoint "${tmp}") + mkdir -p rootfs + if ! enroot-nsenter ${unpriv:+--user} --mount --remap-root \ + bash -c "mount --make-rprivate / && mount -t overlay overlay -o lowerdir=0:$(seq -s: 1 "${layer_count}") rootfs && + tar --numeric-owner -C rootfs/ --mode=u-s,g-s -cpf - . | tar --numeric-owner -C '${mountpoint}/' -xpf -"; then + zfs destroy -r "${tmp}" 2> /dev/null || : + common::err "Failed to merge Docker layers into ZFS template" + fi + zfs rename "${tmp}" "${template}" + zfs snapshot "${snap}" + zfs set readonly=on "${template}" + else + # Lost the race or stale .tmp — wait for @pristine. + while ! zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; do + sleep 1 + ((i++ < 600)) || common::err "Timed out waiting for Docker template: ${template}" + done + fi + + zfs::clone_container "${template}" "${name}" +}