Plan C: .zfs image format (zfs send streams)#7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sodre merged 6 commits into from
Apr 29, 2026
Merged

Plan C: .zfs image format (zfs send streams) #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
baf35ef
Add zfs::ensure_template_from_stream and zfs::send_stream helpers
sodre Apr 29, 2026
9f8d8fe
Dispatch runtime::create on .sqsh vs .zfs extension
sodre Apr 29, 2026
aeff6d0
Add --format=zfs to enroot export
sodre Apr 29, 2026
502531f
Mark Plan C (.zfs image format) as implemented
sodre Apr 29, 2026
ced1c56
Restore upstream comments dropped by earlier ZFS refactors
sodre Apr 29, 2026
4775c8d
Move Plan C ZFS-side helpers into storage_zfs.sh
sodre Apr 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@ When debugging container behavior, the order is: image `/etc/{rc,fstab,environme
## Active design proposals

- **`doc/zfs.md`** — optional ZFS storage backend (`ENROOT_STORAGE_BACKEND=zfs`). Replaces `unsquashfs`-per-create with extract-once-then-`zfs clone`. Adds a `.zfs` (zfs send stream) image format and a `zfs://host/NAME` transport scheme alongside today's `.sqsh`. Introduces a shared template cache with a live/warm/cold lifecycle (knobs: `ENROOT_TEMPLATE_WARM_SECONDS`, `ENROOT_TEMPLATE_PRESSURE_THRESHOLD`; eviction is implicit on `create`, no daemon, no `enroot gc` command). Default backend (`dir`) is unchanged.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, Fmerged; B in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5)); C, D are still design-only.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, F, B merged; C in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5), [#7](https://github.com/zeroae/enroot/pull/7)); D is still design-only.

## Conventions

Expand Down
2 changes: 1 addition & 1 deletion doc/zfs.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
# ZFS storage backend

This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. The remaining transports (`.zfs` file format, `zfs://` URI) are tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.
This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), C (`.zfs` image format), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. `enroot create` accepts both `.sqsh` and `.zfs` (zfs send stream) inputs; `enroot export --format=zfs` produces the latter. The remaining transport (`zfs://` URI) is tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.

## Motivation

Expand Down
17 changes: 14 additions & 3 deletions enroot.in
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -170,8 +170,9 @@ enroot::usage() {
Create a container image from a container root filesystem.

Options:
-o, --output Name of the output image file (defaults to "NAME.sqsh")
-o, --output Name of the output image file (defaults to "NAME.sqsh" or "NAME.zfs")
-f, --force Overwrite an existing container image
--format Output format: "sqsh" (default) or "zfs" (zfs send stream; requires ZFS backend)
EOF
;;
import)
Expand DownExpand Up@@ -444,7 +445,7 @@ enroot::load() {
}

enroot::export() {
local name= filename=
local name= filename= format=sqsh

while [ $# -gt 0 ]; do
case "$1" in
Expand All@@ -462,6 +463,16 @@ enroot::export() {
filename="${1#*=}"
shift
;;
--format)
[ -z "${2-}" ] && enroot::usage export 1
format="$2"
shift 2
;;
--format=*)
[ -z "${1#*=}" ] && enroot::usage export 1
format="${1#*=}"
shift
;;
-h|--help)
enroot::usage export 0 ;;
--)
Expand All@@ -477,7 +488,7 @@ enroot::export() {
fi
name="$1"

runtime::export "${name}" "${filename}"
runtime::export "${name}" "${filename}" "${format}"
}

enroot::create() {
Expand Down
93 changes: 68 additions & 25 deletions src/runtime.sh
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -437,23 +437,37 @@ runtime::create() {
if [ ! -f "${image}" ]; then
common::err "No such file or directory: ${image}"
fi
if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
common::err "Invalid image format: ${image}"
fi

# Resolve the container rootfs name.
if [ -z "${rootfs}" ]; then
rootfs=$(basename "${image%.sqsh}")
fi
if [[ "${rootfs}" == */* ]]; then
common::err "Invalid argument: ${rootfs}"
fi

if zfs::enabled; then
runtime::_create_zfs "${image}" "${rootfs}"
else
runtime::_create_dir "${image}" "${rootfs}"
fi
case "${image}" in
*.zfs)
if ! zfs::enabled; then
common::err ".zfs images require ENROOT_STORAGE_BACKEND=zfs"
fi
if [ -z "${rootfs}" ]; then
rootfs=$(basename "${image%.zfs}")
fi
if [[ "${rootfs}" == */* ]]; then
common::err "Invalid argument: ${rootfs}"
fi
zfs::create_from_stream "${image}" "${rootfs}"
;;
*)
if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
common::err "Invalid image format: ${image} (expected .sqsh or .zfs)"
fi
if [ -z "${rootfs}" ]; then
rootfs=$(basename "${image%.sqsh}")
fi
if [[ "${rootfs}" == */* ]]; then
common::err "Invalid argument: ${rootfs}"
fi
if zfs::enabled; then
runtime::_create_zfs "${image}" "${rootfs}"
else
runtime::_create_dir "${image}" "${rootfs}"
fi
;;
esac
}

runtime::_create_dir() {
Expand All@@ -462,6 +476,7 @@ runtime::_create_dir() {

common::checkcmd unsquashfs find

# Resolve the container rootfs path.
rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
if [ -e "${rootfs}" ]; then
if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
Expand All@@ -471,6 +486,7 @@ runtime::_create_dir() {
fi
fi

# Extract the container rootfs from the image.
common::log INFO "Extracting squashfs filesystem..." NL
# XXX: https://github.com/NVIDIA/enroot/issues/90
[ $(ulimit -n) -gt $((2**26)) ] && ulimit -n $((2**26))
Expand DownExpand Up@@ -549,19 +565,41 @@ runtime::load() {
}

runtime::export() {
local rootfs="$1" filename="$2"
local exclude=()

common::checkcmd mksquashfs
local rootfs_name="$1" filename="$2" format="${3:-sqsh}"

# Resolve the container rootfs path.
if [ -z "${rootfs}" ]; then
if [ -z "${rootfs_name}" ]; then
common::err "Invalid argument"
fi
if [[ "${rootfs}" == */* ]]; then
common::err "Invalid argument: ${rootfs}"
if [[ "${rootfs_name}" == */* ]]; then
common::err "Invalid argument: ${rootfs_name}"
fi

case "${format}" in
sqsh) ;;
zfs)
if ! zfs::enabled; then
common::err "--format=zfs requires ENROOT_STORAGE_BACKEND=zfs"
fi
;;
*) common::err "Invalid format: ${format}" ;;
esac

if [ "${format}" = "sqsh" ]; then
runtime::_export_sqsh "${rootfs_name}" "${filename}"
else
zfs::export_to_file "${rootfs_name}" "${filename}"
fi
rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs}")
}

runtime::_export_sqsh() {
local -r rootfs_name="$1"
local filename="$2"
local rootfs exclude=()

common::checkcmd mksquashfs

# Resolve the container rootfs path.
rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
if [ ! -d "${rootfs}" ]; then
common::err "No such file or directory: ${rootfs}"
fi
Expand DownExpand Up@@ -657,6 +695,7 @@ runtime::list() {
runtime::remove() {
local rootfs_name="$1"

# Resolve the container rootfs path.
if [ -z "${rootfs_name}" ]; then
common::err "Invalid argument"
fi
Expand All@@ -678,6 +717,8 @@ runtime::_remove_dir() {
if [ ! -d "${rootfs}" ]; then
common::err "No such file or directory: ${rootfs}"
fi

# Remove the rootfs specified after asking for confirmation.
if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
fi
Expand All@@ -690,6 +731,8 @@ runtime::_remove_zfs() {
local -r rootfs_name="$1"
local rootfs
rootfs="${ENROOT_DATA_PATH}/${rootfs_name}"

# Remove the rootfs specified after asking for confirmation.
if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
fi
Expand Down
118 changes: 118 additions & 0 deletions src/storage_zfs.sh
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -262,6 +262,124 @@ zfs::container_check() {
fi
}

# Materializes a ZFS stream file into a template (cached by file sha) and
# clones it as the user's named container. Counterpart of zfs::ensure_template
# + zfs::clone_container for the .sqsh path; this is called from runtime::create
# when the input image has a .zfs extension.
zfs::create_from_stream() {
local -r image="$1" name="$2"
local sha template

zfs::checkenv
sha=$(zfs::image_sha256 "${image}")
template=$(zfs::ensure_template_from_stream "${image}" "${sha}")
zfs::clone_container "${template}" "${name}"
}

# Exports a clone's @pristine snapshot as a zfs send stream file. Owns
# filename defaulting and the file-already-exists guard so runtime::export's
# ZFS branch is a single dispatch call.
zfs::export_to_file() {
local -r name="$1"
local filename="$2"

if [ -z "${filename}" ]; then
filename="${name}.zfs"
fi
filename=$(common::realpath "${filename}")
if [ -e "${filename}" ]; then
if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
common::err "File already exists: ${filename}"
else
rm -f "${filename}"
fi
fi

common::log INFO "Creating zfs send stream..." NL
zfs::send_stream "${name}" "${filename}"
}

# Materializes a template from a zfs send stream file. The cache key is the
# sha256 of the stream file (same scheme as the .sqsh path). Atomic via a
# .tmp dataset; integrates with the same eviction sweep as ensure_template.
zfs::ensure_template_from_stream() {
local -r stream="$1" sha="$2"
local -r store=$(zfs::store_dataset)
local -r template="${store}/${zfs_template_subdir}/${sha}"
local -r tmp="${template}.tmp"
local -r snap="${template}@${zfs_pristine_snap}"
local i timeout=600

zfs::sweep_templates

# Fast path: already cached.
if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
zfs::touch_template "${template}"
printf "%s" "${template}"
return
fi

# Ensure the templates parent exists before receive (zfs receive does not
# auto-create parents).
zfs create -p "${store}/${zfs_template_subdir}" 2> /dev/null || :

if zfs receive -F "${tmp}" < "${stream}" 2> /dev/null; then
# The received dataset brings its own snapshot. Rename the dataset to
# the final template name; if the recv'd snapshot wasn't already named
# @pristine, alias it.
zfs rename "${tmp}" "${template}"
if ! zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
local recvd_snap
recvd_snap=$(zfs list -H -t snapshot -o name -r -d 1 "${template}" | head -1)
[ -n "${recvd_snap}" ] && zfs rename "${recvd_snap}" "${snap}"
fi
zfs set readonly=on "${template}"
zfs::touch_template "${template}"
printf "%s" "${template}"
return
fi

# Receive failed. Clean our orphan .tmp (if any) and wait for another
# writer's @pristine.
zfs destroy -r "${tmp}" 2> /dev/null || :
for ((i = 0; i < timeout; i++)); do
if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
printf "%s" "${template}"
return
fi
sleep 1
done
common::err "Timed out waiting for stream receive: ${template}"
}

# Sends a clone's @pristine snapshot (or a fresh snapshot if the container is
# not a clone) to stdout. Used by --format=zfs export.
zfs::send_stream() {
local -r name="$1" filename="$2"
local -r store=$(zfs::store_dataset)
local -r target="${store}/${name}"
local origin

if ! zfs list -H "${target}" > /dev/null 2>&1; then
common::err "No such container: ${name}"
fi

origin=$(zfs get -H -o value origin "${target}")
if [ -z "${origin}" ] || [ "${origin}" = "-" ]; then
# Not a clone — must take a fresh snapshot of the live dataset.
local snap="${target}@enroot-export-$$"
zfs snapshot "${snap}"
if ! zfs send "${snap}" > "${filename}"; then
zfs destroy "${snap}" 2> /dev/null || :
common::err "Failed to send stream for ${name}"
fi
zfs destroy "${snap}" 2> /dev/null || :
else
zfs send "${origin}" > "${filename}" \
|| common::err "Failed to send stream for ${name}"
fi
}

# Materializes the merged Docker rootfs into a ZFS template (cached by
# cache_key) and clones it as the user's named container. Designed to be
# called from docker::load AFTER docker::_prepare_layers has populated the
Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Plan C: .zfs image format (zfs send streams)#7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

sodre
merged 6 commits into

from

Apr 29, 2026
 Merged


Plan C: .zfs image format (zfs send streams)
#7
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
baf35ef

Add zfs::ensure_template_from_stream and zfs::send_stream helpers


sodre Apr 29, 2026
9f8d8fe

Dispatch runtime::create on .sqsh vs .zfs extension


sodre Apr 29, 2026
aeff6d0

Add --format=zfs to enroot export


sodre Apr 29, 2026
502531f

Mark Plan C (.zfs image format) as implemented


sodre Apr 29, 2026
ced1c56

Restore upstream comments dropped by earlier ZFS refactors


sodre Apr 29, 2026
4775c8d

Move Plan C ZFS-side helpers into storage_zfs.sh


sodre Apr 29, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
CLAUDE.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@ When debugging container behavior, the order is: image `/etc/{rc,fstab,environme
## Active design proposals

- **`doc/zfs.md`** — optional ZFS storage backend (`ENROOT_STORAGE_BACKEND=zfs`). Replaces `unsquashfs`-per-create with extract-once-then-`zfs clone`. Adds a `.zfs` (zfs send stream) image format and a `zfs://host/NAME` transport scheme alongside today's `.sqsh`. Introduces a shared template cache with a live/warm/cold lifecycle (knobs: `ENROOT_TEMPLATE_WARM_SECONDS`, `ENROOT_TEMPLATE_PRESSURE_THRESHOLD`; eviction is implicit on `create`, no daemon, no `enroot gc` command). Default backend (`dir`) is unchanged.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, Fmerged; B in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5)); C, D are still design-only.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, F, B merged; C in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5), [#7](https://github.com/zeroae/enroot/pull/7)); D is still design-only.

## Conventions

Expand Down

2 changes: 1 addition & 1 deletion
doc/zfs.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
# ZFS storage backend

This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. The remaining transports (`.zfs` file format, `zfs://` URI) are tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.
This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), C (`.zfs` image format), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. `enroot create` accepts both `.sqsh` and `.zfs` (zfs send stream) inputs; `enroot export --format=zfs` produces the latter. The remaining transport (`zfs://` URI) is tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.

## Motivation

Expand Down

17 changes: 14 additions & 3 deletions
enroot.in

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -170,8 +170,9 @@ enroot::usage() {
 Create a container image from a container root filesystem.

 Options:
 -o, --output Name of the output image file (defaults to "NAME.sqsh")
 -o, --output Name of the output image file (defaults to "NAME.sqsh" or "NAME.zfs")
 -f, --force Overwrite an existing container image
 --format Output format: "sqsh" (default) or "zfs" (zfs send stream; requires ZFS backend)
 EOF
 ;;
 import)
Expand DownExpand Up@@ -444,7 +445,7 @@ enroot::load() {
}

enroot::export() {
 local name= filename=
 local name= filename= format=sqsh

 while [ $# -gt 0 ]; do
 case "$1" in
Expand All@@ -462,6 +463,16 @@ enroot::export() {
 filename="${1#*=}"
 shift
 ;;
 --format)
 [ -z "${2-}" ] && enroot::usage export 1
 format="$2"
 shift 2
 ;;
 --format=*)
 [ -z "${1#*=}" ] && enroot::usage export 1
 format="${1#*=}"
 shift
 ;;
 -h|--help)
 enroot::usage export 0 ;;
 --)
Expand All@@ -477,7 +488,7 @@ enroot::export() {
 fi
 name="$1"

 runtime::export "${name}" "${filename}"
 runtime::export "${name}" "${filename}" "${format}"
}

enroot::create() {
Expand Down

93 changes: 68 additions & 25 deletions
src/runtime.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -437,23 +437,37 @@ runtime::create() {
 if [ ! -f "${image}" ]; then
 common::err "No such file or directory: ${image}"
 fi
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image}"
 fi

 # Resolve the container rootfs name.
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi

 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 case "${image}" in
 *.zfs)
 if ! zfs::enabled; then
 common::err ".zfs images require ENROOT_STORAGE_BACKEND=zfs"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.zfs}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 zfs::create_from_stream "${image}" "${rootfs}"
 ;;
 *)
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image} (expected .sqsh or .zfs)"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 ;;
 esac
}

runtime::_create_dir() {
Expand All@@ -462,6 +476,7 @@ runtime::_create_dir() {

 common::checkcmd unsquashfs find

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ -e "${rootfs}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
Expand All@@ -471,6 +486,7 @@ runtime::_create_dir() {
 fi
 fi

 # Extract the container rootfs from the image.
 common::log INFO "Extracting squashfs filesystem..." NL
 # XXX: https://github.com/NVIDIA/enroot/issues/90
 [ $(ulimit -n) -gt $((2**26)) ] && ulimit -n $((2**26))
Expand DownExpand Up@@ -549,19 +565,41 @@ runtime::load() {
}

runtime::export() {
 local rootfs="$1" filename="$2"
 local exclude=()

 common::checkcmd mksquashfs
 local rootfs_name="$1" filename="$2" format="${3:-sqsh}"

 # Resolve the container rootfs path.
 if [ -z "${rootfs}" ]; then
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 if [[ "${rootfs_name}" == */* ]]; then
 common::err "Invalid argument: ${rootfs_name}"
 fi

 case "${format}" in
 sqsh) ;;
 zfs)
 if ! zfs::enabled; then
 common::err "--format=zfs requires ENROOT_STORAGE_BACKEND=zfs"
 fi
 ;;
 *) common::err "Invalid format: ${format}" ;;
 esac

 if [ "${format}" = "sqsh" ]; then
 runtime::_export_sqsh "${rootfs_name}" "${filename}"
 else
 zfs::export_to_file "${rootfs_name}" "${filename}"
 fi
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs}")
}

runtime::_export_sqsh() {
 local -r rootfs_name="$1"
 local filename="$2"
 local rootfs exclude=()

 common::checkcmd mksquashfs

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi
Expand DownExpand Up@@ -657,6 +695,7 @@ runtime::list() {
runtime::remove() {
 local rootfs_name="$1"

 # Resolve the container rootfs path.
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
Expand All@@ -678,6 +717,8 @@ runtime::_remove_dir() {
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand All@@ -690,6 +731,8 @@ runtime::_remove_zfs() {
 local -r rootfs_name="$1"
 local rootfs
 rootfs="${ENROOT_DATA_PATH}/${rootfs_name}"

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand Down

118 changes: 118 additions & 0 deletions
src/storage_zfs.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -262,6 +262,124 @@ zfs::container_check() {
 fi
}

# Materializes a ZFS stream file into a template (cached by file sha) and
# clones it as the user's named container. Counterpart of zfs::ensure_template
# + zfs::clone_container for the .sqsh path; this is called from runtime::create
# when the input image has a .zfs extension.
zfs::create_from_stream() {
 local -r image="$1" name="$2"
 local sha template

 zfs::checkenv
 sha=$(zfs::image_sha256 "${image}")
 template=$(zfs::ensure_template_from_stream "${image}" "${sha}")
 zfs::clone_container "${template}" "${name}"
}

# Exports a clone's @pristine snapshot as a zfs send stream file. Owns
# filename defaulting and the file-already-exists guard so runtime::export's
# ZFS branch is a single dispatch call.
zfs::export_to_file() {
 local -r name="$1"
 local filename="$2"

 if [ -z "${filename}" ]; then
 filename="${name}.zfs"
 fi
 filename=$(common::realpath "${filename}")
 if [ -e "${filename}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 common::err "File already exists: ${filename}"
 else
 rm -f "${filename}"
 fi
 fi

 common::log INFO "Creating zfs send stream..." NL
 zfs::send_stream "${name}" "${filename}"
}

# Materializes a template from a zfs send stream file. The cache key is the
# sha256 of the stream file (same scheme as the .sqsh path). Atomic via a
# .tmp dataset; integrates with the same eviction sweep as ensure_template.
zfs::ensure_template_from_stream() {
 local -r stream="$1" sha="$2"
 local -r store=$(zfs::store_dataset)
 local -r template="${store}/${zfs_template_subdir}/${sha}"
 local -r tmp="${template}.tmp"
 local -r snap="${template}@${zfs_pristine_snap}"
 local i timeout=600

 zfs::sweep_templates

 # Fast path: already cached.
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Ensure the templates parent exists before receive (zfs receive does not
 # auto-create parents).
 zfs create -p "${store}/${zfs_template_subdir}" 2> /dev/null || :

 if zfs receive -F "${tmp}" < "${stream}" 2> /dev/null; then
 # The received dataset brings its own snapshot. Rename the dataset to
 # the final template name; if the recv'd snapshot wasn't already named
 # @pristine, alias it.
 zfs rename "${tmp}" "${template}"
 if ! zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 local recvd_snap
 recvd_snap=$(zfs list -H -t snapshot -o name -r -d 1 "${template}" | head -1)
 [ -n "${recvd_snap}" ] && zfs rename "${recvd_snap}" "${snap}"
 fi
 zfs set readonly=on "${template}"
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Receive failed. Clean our orphan .tmp (if any) and wait for another
 # writer's @pristine.
 zfs destroy -r "${tmp}" 2> /dev/null || :
 for ((i = 0; i < timeout; i++)); do
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 printf "%s" "${template}"
 return
 fi
 sleep 1
 done
 common::err "Timed out waiting for stream receive: ${template}"
}

# Sends a clone's @pristine snapshot (or a fresh snapshot if the container is
# not a clone) to stdout. Used by --format=zfs export.
zfs::send_stream() {
 local -r name="$1" filename="$2"
 local -r store=$(zfs::store_dataset)
 local -r target="${store}/${name}"
 local origin

 if ! zfs list -H "${target}" > /dev/null 2>&1; then
 common::err "No such container: ${name}"
 fi

 origin=$(zfs get -H -o value origin "${target}")
 if [ -z "${origin}" ] || [ "${origin}" = "-" ]; then
 # Not a clone — must take a fresh snapshot of the live dataset.
 local snap="${target}@enroot-export-$$"
 zfs snapshot "${snap}"
 if ! zfs send "${snap}" > "${filename}"; then
 zfs destroy "${snap}" 2> /dev/null || :
 common::err "Failed to send stream for ${name}"
 fi
 zfs destroy "${snap}" 2> /dev/null || :
 else
 zfs send "${origin}" > "${filename}" \
 || common::err "Failed to send stream for ${name}"
 fi
}

# Materializes the merged Docker rootfs into a ZFS template (cached by
# cache_key) and clones it as the user's named container. Designed to be
# called from docker::load AFTER docker::_prepare_layers has populated the
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Plan C: .zfs image format (zfs send streams)#7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

sodre
merged 6 commits into

from

Apr 29, 2026
 Merged


Plan C: .zfs image format (zfs send streams)
#7
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
baf35ef

Add zfs::ensure_template_from_stream and zfs::send_stream helpers


sodre Apr 29, 2026
9f8d8fe

Dispatch runtime::create on .sqsh vs .zfs extension


sodre Apr 29, 2026
aeff6d0

Add --format=zfs to enroot export


sodre Apr 29, 2026
502531f

Mark Plan C (.zfs image format) as implemented


sodre Apr 29, 2026
ced1c56

Restore upstream comments dropped by earlier ZFS refactors


sodre Apr 29, 2026
4775c8d

Move Plan C ZFS-side helpers into storage_zfs.sh


sodre Apr 29, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
CLAUDE.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@ When debugging container behavior, the order is: image `/etc/{rc,fstab,environme
## Active design proposals

- **`doc/zfs.md`** — optional ZFS storage backend (`ENROOT_STORAGE_BACKEND=zfs`). Replaces `unsquashfs`-per-create with extract-once-then-`zfs clone`. Adds a `.zfs` (zfs send stream) image format and a `zfs://host/NAME` transport scheme alongside today's `.sqsh`. Introduces a shared template cache with a live/warm/cold lifecycle (knobs: `ENROOT_TEMPLATE_WARM_SECONDS`, `ENROOT_TEMPLATE_PRESSURE_THRESHOLD`; eviction is implicit on `create`, no daemon, no `enroot gc` command). Default backend (`dir`) is unchanged.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, Fmerged; B in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5)); C, D are still design-only.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, F, B merged; C in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5), [#7](https://github.com/zeroae/enroot/pull/7)); D is still design-only.

## Conventions

Expand Down

2 changes: 1 addition & 1 deletion
doc/zfs.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
# ZFS storage backend

This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. The remaining transports (`.zfs` file format, `zfs://` URI) are tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.
This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), C (`.zfs` image format), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. `enroot create` accepts both `.sqsh` and `.zfs` (zfs send stream) inputs; `enroot export --format=zfs` produces the latter. The remaining transport (`zfs://` URI) is tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.

## Motivation

Expand Down

17 changes: 14 additions & 3 deletions
enroot.in

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -170,8 +170,9 @@ enroot::usage() {
 Create a container image from a container root filesystem.

 Options:
 -o, --output Name of the output image file (defaults to "NAME.sqsh")
 -o, --output Name of the output image file (defaults to "NAME.sqsh" or "NAME.zfs")
 -f, --force Overwrite an existing container image
 --format Output format: "sqsh" (default) or "zfs" (zfs send stream; requires ZFS backend)
 EOF
 ;;
 import)
Expand DownExpand Up@@ -444,7 +445,7 @@ enroot::load() {
}

enroot::export() {
 local name= filename=
 local name= filename= format=sqsh

 while [ $# -gt 0 ]; do
 case "$1" in
Expand All@@ -462,6 +463,16 @@ enroot::export() {
 filename="${1#*=}"
 shift
 ;;
 --format)
 [ -z "${2-}" ] && enroot::usage export 1
 format="$2"
 shift 2
 ;;
 --format=*)
 [ -z "${1#*=}" ] && enroot::usage export 1
 format="${1#*=}"
 shift
 ;;
 -h|--help)
 enroot::usage export 0 ;;
 --)
Expand All@@ -477,7 +488,7 @@ enroot::export() {
 fi
 name="$1"

 runtime::export "${name}" "${filename}"
 runtime::export "${name}" "${filename}" "${format}"
}

enroot::create() {
Expand Down

93 changes: 68 additions & 25 deletions
src/runtime.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -437,23 +437,37 @@ runtime::create() {
 if [ ! -f "${image}" ]; then
 common::err "No such file or directory: ${image}"
 fi
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image}"
 fi

 # Resolve the container rootfs name.
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi

 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 case "${image}" in
 *.zfs)
 if ! zfs::enabled; then
 common::err ".zfs images require ENROOT_STORAGE_BACKEND=zfs"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.zfs}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 zfs::create_from_stream "${image}" "${rootfs}"
 ;;
 *)
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image} (expected .sqsh or .zfs)"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 ;;
 esac
}

runtime::_create_dir() {
Expand All@@ -462,6 +476,7 @@ runtime::_create_dir() {

 common::checkcmd unsquashfs find

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ -e "${rootfs}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
Expand All@@ -471,6 +486,7 @@ runtime::_create_dir() {
 fi
 fi

 # Extract the container rootfs from the image.
 common::log INFO "Extracting squashfs filesystem..." NL
 # XXX: https://github.com/NVIDIA/enroot/issues/90
 [ $(ulimit -n) -gt $((2**26)) ] && ulimit -n $((2**26))
Expand DownExpand Up@@ -549,19 +565,41 @@ runtime::load() {
}

runtime::export() {
 local rootfs="$1" filename="$2"
 local exclude=()

 common::checkcmd mksquashfs
 local rootfs_name="$1" filename="$2" format="${3:-sqsh}"

 # Resolve the container rootfs path.
 if [ -z "${rootfs}" ]; then
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 if [[ "${rootfs_name}" == */* ]]; then
 common::err "Invalid argument: ${rootfs_name}"
 fi

 case "${format}" in
 sqsh) ;;
 zfs)
 if ! zfs::enabled; then
 common::err "--format=zfs requires ENROOT_STORAGE_BACKEND=zfs"
 fi
 ;;
 *) common::err "Invalid format: ${format}" ;;
 esac

 if [ "${format}" = "sqsh" ]; then
 runtime::_export_sqsh "${rootfs_name}" "${filename}"
 else
 zfs::export_to_file "${rootfs_name}" "${filename}"
 fi
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs}")
}

runtime::_export_sqsh() {
 local -r rootfs_name="$1"
 local filename="$2"
 local rootfs exclude=()

 common::checkcmd mksquashfs

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi
Expand DownExpand Up@@ -657,6 +695,7 @@ runtime::list() {
runtime::remove() {
 local rootfs_name="$1"

 # Resolve the container rootfs path.
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
Expand All@@ -678,6 +717,8 @@ runtime::_remove_dir() {
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand All@@ -690,6 +731,8 @@ runtime::_remove_zfs() {
 local -r rootfs_name="$1"
 local rootfs
 rootfs="${ENROOT_DATA_PATH}/${rootfs_name}"

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand Down

118 changes: 118 additions & 0 deletions
src/storage_zfs.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -262,6 +262,124 @@ zfs::container_check() {
 fi
}

# Materializes a ZFS stream file into a template (cached by file sha) and
# clones it as the user's named container. Counterpart of zfs::ensure_template
# + zfs::clone_container for the .sqsh path; this is called from runtime::create
# when the input image has a .zfs extension.
zfs::create_from_stream() {
 local -r image="$1" name="$2"
 local sha template

 zfs::checkenv
 sha=$(zfs::image_sha256 "${image}")
 template=$(zfs::ensure_template_from_stream "${image}" "${sha}")
 zfs::clone_container "${template}" "${name}"
}

# Exports a clone's @pristine snapshot as a zfs send stream file. Owns
# filename defaulting and the file-already-exists guard so runtime::export's
# ZFS branch is a single dispatch call.
zfs::export_to_file() {
 local -r name="$1"
 local filename="$2"

 if [ -z "${filename}" ]; then
 filename="${name}.zfs"
 fi
 filename=$(common::realpath "${filename}")
 if [ -e "${filename}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 common::err "File already exists: ${filename}"
 else
 rm -f "${filename}"
 fi
 fi

 common::log INFO "Creating zfs send stream..." NL
 zfs::send_stream "${name}" "${filename}"
}

# Materializes a template from a zfs send stream file. The cache key is the
# sha256 of the stream file (same scheme as the .sqsh path). Atomic via a
# .tmp dataset; integrates with the same eviction sweep as ensure_template.
zfs::ensure_template_from_stream() {
 local -r stream="$1" sha="$2"
 local -r store=$(zfs::store_dataset)
 local -r template="${store}/${zfs_template_subdir}/${sha}"
 local -r tmp="${template}.tmp"
 local -r snap="${template}@${zfs_pristine_snap}"
 local i timeout=600

 zfs::sweep_templates

 # Fast path: already cached.
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Ensure the templates parent exists before receive (zfs receive does not
 # auto-create parents).
 zfs create -p "${store}/${zfs_template_subdir}" 2> /dev/null || :

 if zfs receive -F "${tmp}" < "${stream}" 2> /dev/null; then
 # The received dataset brings its own snapshot. Rename the dataset to
 # the final template name; if the recv'd snapshot wasn't already named
 # @pristine, alias it.
 zfs rename "${tmp}" "${template}"
 if ! zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 local recvd_snap
 recvd_snap=$(zfs list -H -t snapshot -o name -r -d 1 "${template}" | head -1)
 [ -n "${recvd_snap}" ] && zfs rename "${recvd_snap}" "${snap}"
 fi
 zfs set readonly=on "${template}"
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Receive failed. Clean our orphan .tmp (if any) and wait for another
 # writer's @pristine.
 zfs destroy -r "${tmp}" 2> /dev/null || :
 for ((i = 0; i < timeout; i++)); do
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 printf "%s" "${template}"
 return
 fi
 sleep 1
 done
 common::err "Timed out waiting for stream receive: ${template}"
}

# Sends a clone's @pristine snapshot (or a fresh snapshot if the container is
# not a clone) to stdout. Used by --format=zfs export.
zfs::send_stream() {
 local -r name="$1" filename="$2"
 local -r store=$(zfs::store_dataset)
 local -r target="${store}/${name}"
 local origin

 if ! zfs list -H "${target}" > /dev/null 2>&1; then
 common::err "No such container: ${name}"
 fi

 origin=$(zfs get -H -o value origin "${target}")
 if [ -z "${origin}" ] || [ "${origin}" = "-" ]; then
 # Not a clone — must take a fresh snapshot of the live dataset.
 local snap="${target}@enroot-export-$$"
 zfs snapshot "${snap}"
 if ! zfs send "${snap}" > "${filename}"; then
 zfs destroy "${snap}" 2> /dev/null || :
 common::err "Failed to send stream for ${name}"
 fi
 zfs destroy "${snap}" 2> /dev/null || :
 else
 zfs send "${origin}" > "${filename}" \
 || common::err "Failed to send stream for ${name}"
 fi
}

# Materializes the merged Docker rootfs into a ZFS template (cached by
# cache_key) and clones it as the user's named container. Designed to be
# called from docker::load AFTER docker::_prepare_layers has populated the
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Plan C: .zfs image format (zfs send streams)#7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

sodre
merged 6 commits into

from

Apr 29, 2026
 Merged


Plan C: .zfs image format (zfs send streams)
#7
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
baf35ef

Add zfs::ensure_template_from_stream and zfs::send_stream helpers


sodre Apr 29, 2026
9f8d8fe

Dispatch runtime::create on .sqsh vs .zfs extension


sodre Apr 29, 2026
aeff6d0

Add --format=zfs to enroot export


sodre Apr 29, 2026
502531f

Mark Plan C (.zfs image format) as implemented


sodre Apr 29, 2026
ced1c56

Restore upstream comments dropped by earlier ZFS refactors


sodre Apr 29, 2026
4775c8d

Move Plan C ZFS-side helpers into storage_zfs.sh


sodre Apr 29, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
CLAUDE.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@ When debugging container behavior, the order is: image `/etc/{rc,fstab,environme
## Active design proposals

- **`doc/zfs.md`** — optional ZFS storage backend (`ENROOT_STORAGE_BACKEND=zfs`). Replaces `unsquashfs`-per-create with extract-once-then-`zfs clone`. Adds a `.zfs` (zfs send stream) image format and a `zfs://host/NAME` transport scheme alongside today's `.sqsh`. Introduces a shared template cache with a live/warm/cold lifecycle (knobs: `ENROOT_TEMPLATE_WARM_SECONDS`, `ENROOT_TEMPLATE_PRESSURE_THRESHOLD`; eviction is implicit on `create`, no daemon, no `enroot gc` command). Default backend (`dir`) is unchanged.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, Fmerged; B in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5)); C, D are still design-only.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, F, B merged; C in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5), [#7](https://github.com/zeroae/enroot/pull/7)); D is still design-only.

## Conventions

Expand Down

2 changes: 1 addition & 1 deletion
doc/zfs.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
# ZFS storage backend

This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. The remaining transports (`.zfs` file format, `zfs://` URI) are tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.
This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), C (`.zfs` image format), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. `enroot create` accepts both `.sqsh` and `.zfs` (zfs send stream) inputs; `enroot export --format=zfs` produces the latter. The remaining transport (`zfs://` URI) is tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.

## Motivation

Expand Down

17 changes: 14 additions & 3 deletions
enroot.in

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -170,8 +170,9 @@ enroot::usage() {
 Create a container image from a container root filesystem.

 Options:
 -o, --output Name of the output image file (defaults to "NAME.sqsh")
 -o, --output Name of the output image file (defaults to "NAME.sqsh" or "NAME.zfs")
 -f, --force Overwrite an existing container image
 --format Output format: "sqsh" (default) or "zfs" (zfs send stream; requires ZFS backend)
 EOF
 ;;
 import)
Expand DownExpand Up@@ -444,7 +445,7 @@ enroot::load() {
}

enroot::export() {
 local name= filename=
 local name= filename= format=sqsh

 while [ $# -gt 0 ]; do
 case "$1" in
Expand All@@ -462,6 +463,16 @@ enroot::export() {
 filename="${1#*=}"
 shift
 ;;
 --format)
 [ -z "${2-}" ] && enroot::usage export 1
 format="$2"
 shift 2
 ;;
 --format=*)
 [ -z "${1#*=}" ] && enroot::usage export 1
 format="${1#*=}"
 shift
 ;;
 -h|--help)
 enroot::usage export 0 ;;
 --)
Expand All@@ -477,7 +488,7 @@ enroot::export() {
 fi
 name="$1"

 runtime::export "${name}" "${filename}"
 runtime::export "${name}" "${filename}" "${format}"
}

enroot::create() {
Expand Down

93 changes: 68 additions & 25 deletions
src/runtime.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -437,23 +437,37 @@ runtime::create() {
 if [ ! -f "${image}" ]; then
 common::err "No such file or directory: ${image}"
 fi
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image}"
 fi

 # Resolve the container rootfs name.
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi

 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 case "${image}" in
 *.zfs)
 if ! zfs::enabled; then
 common::err ".zfs images require ENROOT_STORAGE_BACKEND=zfs"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.zfs}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 zfs::create_from_stream "${image}" "${rootfs}"
 ;;
 *)
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image} (expected .sqsh or .zfs)"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 ;;
 esac
}

runtime::_create_dir() {
Expand All@@ -462,6 +476,7 @@ runtime::_create_dir() {

 common::checkcmd unsquashfs find

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ -e "${rootfs}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
Expand All@@ -471,6 +486,7 @@ runtime::_create_dir() {
 fi
 fi

 # Extract the container rootfs from the image.
 common::log INFO "Extracting squashfs filesystem..." NL
 # XXX: https://github.com/NVIDIA/enroot/issues/90
 [ $(ulimit -n) -gt $((2**26)) ] && ulimit -n $((2**26))
Expand DownExpand Up@@ -549,19 +565,41 @@ runtime::load() {
}

runtime::export() {
 local rootfs="$1" filename="$2"
 local exclude=()

 common::checkcmd mksquashfs
 local rootfs_name="$1" filename="$2" format="${3:-sqsh}"

 # Resolve the container rootfs path.
 if [ -z "${rootfs}" ]; then
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 if [[ "${rootfs_name}" == */* ]]; then
 common::err "Invalid argument: ${rootfs_name}"
 fi

 case "${format}" in
 sqsh) ;;
 zfs)
 if ! zfs::enabled; then
 common::err "--format=zfs requires ENROOT_STORAGE_BACKEND=zfs"
 fi
 ;;
 *) common::err "Invalid format: ${format}" ;;
 esac

 if [ "${format}" = "sqsh" ]; then
 runtime::_export_sqsh "${rootfs_name}" "${filename}"
 else
 zfs::export_to_file "${rootfs_name}" "${filename}"
 fi
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs}")
}

runtime::_export_sqsh() {
 local -r rootfs_name="$1"
 local filename="$2"
 local rootfs exclude=()

 common::checkcmd mksquashfs

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi
Expand DownExpand Up@@ -657,6 +695,7 @@ runtime::list() {
runtime::remove() {
 local rootfs_name="$1"

 # Resolve the container rootfs path.
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
Expand All@@ -678,6 +717,8 @@ runtime::_remove_dir() {
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand All@@ -690,6 +731,8 @@ runtime::_remove_zfs() {
 local -r rootfs_name="$1"
 local rootfs
 rootfs="${ENROOT_DATA_PATH}/${rootfs_name}"

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand Down

118 changes: 118 additions & 0 deletions
src/storage_zfs.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -262,6 +262,124 @@ zfs::container_check() {
 fi
}

# Materializes a ZFS stream file into a template (cached by file sha) and
# clones it as the user's named container. Counterpart of zfs::ensure_template
# + zfs::clone_container for the .sqsh path; this is called from runtime::create
# when the input image has a .zfs extension.
zfs::create_from_stream() {
 local -r image="$1" name="$2"
 local sha template

 zfs::checkenv
 sha=$(zfs::image_sha256 "${image}")
 template=$(zfs::ensure_template_from_stream "${image}" "${sha}")
 zfs::clone_container "${template}" "${name}"
}

# Exports a clone's @pristine snapshot as a zfs send stream file. Owns
# filename defaulting and the file-already-exists guard so runtime::export's
# ZFS branch is a single dispatch call.
zfs::export_to_file() {
 local -r name="$1"
 local filename="$2"

 if [ -z "${filename}" ]; then
 filename="${name}.zfs"
 fi
 filename=$(common::realpath "${filename}")
 if [ -e "${filename}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 common::err "File already exists: ${filename}"
 else
 rm -f "${filename}"
 fi
 fi

 common::log INFO "Creating zfs send stream..." NL
 zfs::send_stream "${name}" "${filename}"
}

# Materializes a template from a zfs send stream file. The cache key is the
# sha256 of the stream file (same scheme as the .sqsh path). Atomic via a
# .tmp dataset; integrates with the same eviction sweep as ensure_template.
zfs::ensure_template_from_stream() {
 local -r stream="$1" sha="$2"
 local -r store=$(zfs::store_dataset)
 local -r template="${store}/${zfs_template_subdir}/${sha}"
 local -r tmp="${template}.tmp"
 local -r snap="${template}@${zfs_pristine_snap}"
 local i timeout=600

 zfs::sweep_templates

 # Fast path: already cached.
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Ensure the templates parent exists before receive (zfs receive does not
 # auto-create parents).
 zfs create -p "${store}/${zfs_template_subdir}" 2> /dev/null || :

 if zfs receive -F "${tmp}" < "${stream}" 2> /dev/null; then
 # The received dataset brings its own snapshot. Rename the dataset to
 # the final template name; if the recv'd snapshot wasn't already named
 # @pristine, alias it.
 zfs rename "${tmp}" "${template}"
 if ! zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 local recvd_snap
 recvd_snap=$(zfs list -H -t snapshot -o name -r -d 1 "${template}" | head -1)
 [ -n "${recvd_snap}" ] && zfs rename "${recvd_snap}" "${snap}"
 fi
 zfs set readonly=on "${template}"
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Receive failed. Clean our orphan .tmp (if any) and wait for another
 # writer's @pristine.
 zfs destroy -r "${tmp}" 2> /dev/null || :
 for ((i = 0; i < timeout; i++)); do
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 printf "%s" "${template}"
 return
 fi
 sleep 1
 done
 common::err "Timed out waiting for stream receive: ${template}"
}

# Sends a clone's @pristine snapshot (or a fresh snapshot if the container is
# not a clone) to stdout. Used by --format=zfs export.
zfs::send_stream() {
 local -r name="$1" filename="$2"
 local -r store=$(zfs::store_dataset)
 local -r target="${store}/${name}"
 local origin

 if ! zfs list -H "${target}" > /dev/null 2>&1; then
 common::err "No such container: ${name}"
 fi

 origin=$(zfs get -H -o value origin "${target}")
 if [ -z "${origin}" ] || [ "${origin}" = "-" ]; then
 # Not a clone — must take a fresh snapshot of the live dataset.
 local snap="${target}@enroot-export-$$"
 zfs snapshot "${snap}"
 if ! zfs send "${snap}" > "${filename}"; then
 zfs destroy "${snap}" 2> /dev/null || :
 common::err "Failed to send stream for ${name}"
 fi
 zfs destroy "${snap}" 2> /dev/null || :
 else
 zfs send "${origin}" > "${filename}" \
 || common::err "Failed to send stream for ${name}"
 fi
}

# Materializes the merged Docker rootfs into a ZFS template (cached by
# cache_key) and clones it as the user's named container. Designed to be
# called from docker::load AFTER docker::_prepare_layers has populated the
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Plan C: .zfs image format (zfs send streams)#7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

sodre
merged 6 commits into

from

Apr 29, 2026
 Merged


Plan C: .zfs image format (zfs send streams)
#7
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
baf35ef

Add zfs::ensure_template_from_stream and zfs::send_stream helpers


sodre Apr 29, 2026
9f8d8fe

Dispatch runtime::create on .sqsh vs .zfs extension


sodre Apr 29, 2026
aeff6d0

Add --format=zfs to enroot export


sodre Apr 29, 2026
502531f

Mark Plan C (.zfs image format) as implemented


sodre Apr 29, 2026
ced1c56

Restore upstream comments dropped by earlier ZFS refactors


sodre Apr 29, 2026
4775c8d

Move Plan C ZFS-side helpers into storage_zfs.sh


sodre Apr 29, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
CLAUDE.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@ When debugging container behavior, the order is: image `/etc/{rc,fstab,environme
## Active design proposals

- **`doc/zfs.md`** — optional ZFS storage backend (`ENROOT_STORAGE_BACKEND=zfs`). Replaces `unsquashfs`-per-create with extract-once-then-`zfs clone`. Adds a `.zfs` (zfs send stream) image format and a `zfs://host/NAME` transport scheme alongside today's `.sqsh`. Introduces a shared template cache with a live/warm/cold lifecycle (knobs: `ENROOT_TEMPLATE_WARM_SECONDS`, `ENROOT_TEMPLATE_PRESSURE_THRESHOLD`; eviction is implicit on `create`, no daemon, no `enroot gc` command). Default backend (`dir`) is unchanged.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, Fmerged; B in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5)); C, D are still design-only.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, F, B merged; C in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5), [#7](https://github.com/zeroae/enroot/pull/7)); D is still design-only.

## Conventions

Expand Down

2 changes: 1 addition & 1 deletion
doc/zfs.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
# ZFS storage backend

This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. The remaining transports (`.zfs` file format, `zfs://` URI) are tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.
This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), C (`.zfs` image format), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. `enroot create` accepts both `.sqsh` and `.zfs` (zfs send stream) inputs; `enroot export --format=zfs` produces the latter. The remaining transport (`zfs://` URI) is tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.

## Motivation

Expand Down

17 changes: 14 additions & 3 deletions
enroot.in

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -170,8 +170,9 @@ enroot::usage() {
 Create a container image from a container root filesystem.

 Options:
 -o, --output Name of the output image file (defaults to "NAME.sqsh")
 -o, --output Name of the output image file (defaults to "NAME.sqsh" or "NAME.zfs")
 -f, --force Overwrite an existing container image
 --format Output format: "sqsh" (default) or "zfs" (zfs send stream; requires ZFS backend)
 EOF
 ;;
 import)
Expand DownExpand Up@@ -444,7 +445,7 @@ enroot::load() {
}

enroot::export() {
 local name= filename=
 local name= filename= format=sqsh

 while [ $# -gt 0 ]; do
 case "$1" in
Expand All@@ -462,6 +463,16 @@ enroot::export() {
 filename="${1#*=}"
 shift
 ;;
 --format)
 [ -z "${2-}" ] && enroot::usage export 1
 format="$2"
 shift 2
 ;;
 --format=*)
 [ -z "${1#*=}" ] && enroot::usage export 1
 format="${1#*=}"
 shift
 ;;
 -h|--help)
 enroot::usage export 0 ;;
 --)
Expand All@@ -477,7 +488,7 @@ enroot::export() {
 fi
 name="$1"

 runtime::export "${name}" "${filename}"
 runtime::export "${name}" "${filename}" "${format}"
}

enroot::create() {
Expand Down

93 changes: 68 additions & 25 deletions
src/runtime.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -437,23 +437,37 @@ runtime::create() {
 if [ ! -f "${image}" ]; then
 common::err "No such file or directory: ${image}"
 fi
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image}"
 fi

 # Resolve the container rootfs name.
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi

 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 case "${image}" in
 *.zfs)
 if ! zfs::enabled; then
 common::err ".zfs images require ENROOT_STORAGE_BACKEND=zfs"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.zfs}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 zfs::create_from_stream "${image}" "${rootfs}"
 ;;
 *)
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image} (expected .sqsh or .zfs)"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 ;;
 esac
}

runtime::_create_dir() {
Expand All@@ -462,6 +476,7 @@ runtime::_create_dir() {

 common::checkcmd unsquashfs find

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ -e "${rootfs}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
Expand All@@ -471,6 +486,7 @@ runtime::_create_dir() {
 fi
 fi

 # Extract the container rootfs from the image.
 common::log INFO "Extracting squashfs filesystem..." NL
 # XXX: https://github.com/NVIDIA/enroot/issues/90
 [ $(ulimit -n) -gt $((2**26)) ] && ulimit -n $((2**26))
Expand DownExpand Up@@ -549,19 +565,41 @@ runtime::load() {
}

runtime::export() {
 local rootfs="$1" filename="$2"
 local exclude=()

 common::checkcmd mksquashfs
 local rootfs_name="$1" filename="$2" format="${3:-sqsh}"

 # Resolve the container rootfs path.
 if [ -z "${rootfs}" ]; then
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 if [[ "${rootfs_name}" == */* ]]; then
 common::err "Invalid argument: ${rootfs_name}"
 fi

 case "${format}" in
 sqsh) ;;
 zfs)
 if ! zfs::enabled; then
 common::err "--format=zfs requires ENROOT_STORAGE_BACKEND=zfs"
 fi
 ;;
 *) common::err "Invalid format: ${format}" ;;
 esac

 if [ "${format}" = "sqsh" ]; then
 runtime::_export_sqsh "${rootfs_name}" "${filename}"
 else
 zfs::export_to_file "${rootfs_name}" "${filename}"
 fi
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs}")
}

runtime::_export_sqsh() {
 local -r rootfs_name="$1"
 local filename="$2"
 local rootfs exclude=()

 common::checkcmd mksquashfs

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi
Expand DownExpand Up@@ -657,6 +695,7 @@ runtime::list() {
runtime::remove() {
 local rootfs_name="$1"

 # Resolve the container rootfs path.
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
Expand All@@ -678,6 +717,8 @@ runtime::_remove_dir() {
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand All@@ -690,6 +731,8 @@ runtime::_remove_zfs() {
 local -r rootfs_name="$1"
 local rootfs
 rootfs="${ENROOT_DATA_PATH}/${rootfs_name}"

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand Down

118 changes: 118 additions & 0 deletions
src/storage_zfs.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -262,6 +262,124 @@ zfs::container_check() {
 fi
}

# Materializes a ZFS stream file into a template (cached by file sha) and
# clones it as the user's named container. Counterpart of zfs::ensure_template
# + zfs::clone_container for the .sqsh path; this is called from runtime::create
# when the input image has a .zfs extension.
zfs::create_from_stream() {
 local -r image="$1" name="$2"
 local sha template

 zfs::checkenv
 sha=$(zfs::image_sha256 "${image}")
 template=$(zfs::ensure_template_from_stream "${image}" "${sha}")
 zfs::clone_container "${template}" "${name}"
}

# Exports a clone's @pristine snapshot as a zfs send stream file. Owns
# filename defaulting and the file-already-exists guard so runtime::export's
# ZFS branch is a single dispatch call.
zfs::export_to_file() {
 local -r name="$1"
 local filename="$2"

 if [ -z "${filename}" ]; then
 filename="${name}.zfs"
 fi
 filename=$(common::realpath "${filename}")
 if [ -e "${filename}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 common::err "File already exists: ${filename}"
 else
 rm -f "${filename}"
 fi
 fi

 common::log INFO "Creating zfs send stream..." NL
 zfs::send_stream "${name}" "${filename}"
}

# Materializes a template from a zfs send stream file. The cache key is the
# sha256 of the stream file (same scheme as the .sqsh path). Atomic via a
# .tmp dataset; integrates with the same eviction sweep as ensure_template.
zfs::ensure_template_from_stream() {
 local -r stream="$1" sha="$2"
 local -r store=$(zfs::store_dataset)
 local -r template="${store}/${zfs_template_subdir}/${sha}"
 local -r tmp="${template}.tmp"
 local -r snap="${template}@${zfs_pristine_snap}"
 local i timeout=600

 zfs::sweep_templates

 # Fast path: already cached.
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Ensure the templates parent exists before receive (zfs receive does not
 # auto-create parents).
 zfs create -p "${store}/${zfs_template_subdir}" 2> /dev/null || :

 if zfs receive -F "${tmp}" < "${stream}" 2> /dev/null; then
 # The received dataset brings its own snapshot. Rename the dataset to
 # the final template name; if the recv'd snapshot wasn't already named
 # @pristine, alias it.
 zfs rename "${tmp}" "${template}"
 if ! zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 local recvd_snap
 recvd_snap=$(zfs list -H -t snapshot -o name -r -d 1 "${template}" | head -1)
 [ -n "${recvd_snap}" ] && zfs rename "${recvd_snap}" "${snap}"
 fi
 zfs set readonly=on "${template}"
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Receive failed. Clean our orphan .tmp (if any) and wait for another
 # writer's @pristine.
 zfs destroy -r "${tmp}" 2> /dev/null || :
 for ((i = 0; i < timeout; i++)); do
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 printf "%s" "${template}"
 return
 fi
 sleep 1
 done
 common::err "Timed out waiting for stream receive: ${template}"
}

# Sends a clone's @pristine snapshot (or a fresh snapshot if the container is
# not a clone) to stdout. Used by --format=zfs export.
zfs::send_stream() {
 local -r name="$1" filename="$2"
 local -r store=$(zfs::store_dataset)
 local -r target="${store}/${name}"
 local origin

 if ! zfs list -H "${target}" > /dev/null 2>&1; then
 common::err "No such container: ${name}"
 fi

 origin=$(zfs get -H -o value origin "${target}")
 if [ -z "${origin}" ] || [ "${origin}" = "-" ]; then
 # Not a clone — must take a fresh snapshot of the live dataset.
 local snap="${target}@enroot-export-$$"
 zfs snapshot "${snap}"
 if ! zfs send "${snap}" > "${filename}"; then
 zfs destroy "${snap}" 2> /dev/null || :
 common::err "Failed to send stream for ${name}"
 fi
 zfs destroy "${snap}" 2> /dev/null || :
 else
 zfs send "${origin}" > "${filename}" \
 || common::err "Failed to send stream for ${name}"
 fi
}

# Materializes the merged Docker rootfs into a ZFS template (cached by
# cache_key) and clones it as the user's named container. Designed to be
# called from docker::load AFTER docker::_prepare_layers has populated the
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Plan C: .zfs image format (zfs send streams)#7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

sodre
merged 6 commits into

from

Apr 29, 2026
 Merged


Plan C: .zfs image format (zfs send streams)
#7
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
baf35ef

Add zfs::ensure_template_from_stream and zfs::send_stream helpers


sodre Apr 29, 2026
9f8d8fe

Dispatch runtime::create on .sqsh vs .zfs extension


sodre Apr 29, 2026
aeff6d0

Add --format=zfs to enroot export


sodre Apr 29, 2026
502531f

Mark Plan C (.zfs image format) as implemented


sodre Apr 29, 2026
ced1c56

Restore upstream comments dropped by earlier ZFS refactors


sodre Apr 29, 2026
4775c8d

Move Plan C ZFS-side helpers into storage_zfs.sh


sodre Apr 29, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
CLAUDE.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@ When debugging container behavior, the order is: image `/etc/{rc,fstab,environme
## Active design proposals

- **`doc/zfs.md`** — optional ZFS storage backend (`ENROOT_STORAGE_BACKEND=zfs`). Replaces `unsquashfs`-per-create with extract-once-then-`zfs clone`. Adds a `.zfs` (zfs send stream) image format and a `zfs://host/NAME` transport scheme alongside today's `.sqsh`. Introduces a shared template cache with a live/warm/cold lifecycle (knobs: `ENROOT_TEMPLATE_WARM_SECONDS`, `ENROOT_TEMPLATE_PRESSURE_THRESHOLD`; eviction is implicit on `create`, no daemon, no `enroot gc` command). Default backend (`dir`) is unchanged.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, Fmerged; B in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5)); C, D are still design-only.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, F, B merged; C in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5), [#7](https://github.com/zeroae/enroot/pull/7)); D is still design-only.

## Conventions

Expand Down

2 changes: 1 addition & 1 deletion
doc/zfs.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
# ZFS storage backend

This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. The remaining transports (`.zfs` file format, `zfs://` URI) are tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.
This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), C (`.zfs` image format), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. `enroot create` accepts both `.sqsh` and `.zfs` (zfs send stream) inputs; `enroot export --format=zfs` produces the latter. The remaining transport (`zfs://` URI) is tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.

## Motivation

Expand Down

17 changes: 14 additions & 3 deletions
enroot.in

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -170,8 +170,9 @@ enroot::usage() {
 Create a container image from a container root filesystem.

 Options:
 -o, --output Name of the output image file (defaults to "NAME.sqsh")
 -o, --output Name of the output image file (defaults to "NAME.sqsh" or "NAME.zfs")
 -f, --force Overwrite an existing container image
 --format Output format: "sqsh" (default) or "zfs" (zfs send stream; requires ZFS backend)
 EOF
 ;;
 import)
Expand DownExpand Up@@ -444,7 +445,7 @@ enroot::load() {
}

enroot::export() {
 local name= filename=
 local name= filename= format=sqsh

 while [ $# -gt 0 ]; do
 case "$1" in
Expand All@@ -462,6 +463,16 @@ enroot::export() {
 filename="${1#*=}"
 shift
 ;;
 --format)
 [ -z "${2-}" ] && enroot::usage export 1
 format="$2"
 shift 2
 ;;
 --format=*)
 [ -z "${1#*=}" ] && enroot::usage export 1
 format="${1#*=}"
 shift
 ;;
 -h|--help)
 enroot::usage export 0 ;;
 --)
Expand All@@ -477,7 +488,7 @@ enroot::export() {
 fi
 name="$1"

 runtime::export "${name}" "${filename}"
 runtime::export "${name}" "${filename}" "${format}"
}

enroot::create() {
Expand Down

93 changes: 68 additions & 25 deletions
src/runtime.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -437,23 +437,37 @@ runtime::create() {
 if [ ! -f "${image}" ]; then
 common::err "No such file or directory: ${image}"
 fi
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image}"
 fi

 # Resolve the container rootfs name.
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi

 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 case "${image}" in
 *.zfs)
 if ! zfs::enabled; then
 common::err ".zfs images require ENROOT_STORAGE_BACKEND=zfs"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.zfs}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 zfs::create_from_stream "${image}" "${rootfs}"
 ;;
 *)
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image} (expected .sqsh or .zfs)"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 ;;
 esac
}

runtime::_create_dir() {
Expand All@@ -462,6 +476,7 @@ runtime::_create_dir() {

 common::checkcmd unsquashfs find

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ -e "${rootfs}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
Expand All@@ -471,6 +486,7 @@ runtime::_create_dir() {
 fi
 fi

 # Extract the container rootfs from the image.
 common::log INFO "Extracting squashfs filesystem..." NL
 # XXX: https://github.com/NVIDIA/enroot/issues/90
 [ $(ulimit -n) -gt $((2**26)) ] && ulimit -n $((2**26))
Expand DownExpand Up@@ -549,19 +565,41 @@ runtime::load() {
}

runtime::export() {
 local rootfs="$1" filename="$2"
 local exclude=()

 common::checkcmd mksquashfs
 local rootfs_name="$1" filename="$2" format="${3:-sqsh}"

 # Resolve the container rootfs path.
 if [ -z "${rootfs}" ]; then
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 if [[ "${rootfs_name}" == */* ]]; then
 common::err "Invalid argument: ${rootfs_name}"
 fi

 case "${format}" in
 sqsh) ;;
 zfs)
 if ! zfs::enabled; then
 common::err "--format=zfs requires ENROOT_STORAGE_BACKEND=zfs"
 fi
 ;;
 *) common::err "Invalid format: ${format}" ;;
 esac

 if [ "${format}" = "sqsh" ]; then
 runtime::_export_sqsh "${rootfs_name}" "${filename}"
 else
 zfs::export_to_file "${rootfs_name}" "${filename}"
 fi
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs}")
}

runtime::_export_sqsh() {
 local -r rootfs_name="$1"
 local filename="$2"
 local rootfs exclude=()

 common::checkcmd mksquashfs

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi
Expand DownExpand Up@@ -657,6 +695,7 @@ runtime::list() {
runtime::remove() {
 local rootfs_name="$1"

 # Resolve the container rootfs path.
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
Expand All@@ -678,6 +717,8 @@ runtime::_remove_dir() {
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand All@@ -690,6 +731,8 @@ runtime::_remove_zfs() {
 local -r rootfs_name="$1"
 local rootfs
 rootfs="${ENROOT_DATA_PATH}/${rootfs_name}"

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand Down

118 changes: 118 additions & 0 deletions
src/storage_zfs.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -262,6 +262,124 @@ zfs::container_check() {
 fi
}

# Materializes a ZFS stream file into a template (cached by file sha) and
# clones it as the user's named container. Counterpart of zfs::ensure_template
# + zfs::clone_container for the .sqsh path; this is called from runtime::create
# when the input image has a .zfs extension.
zfs::create_from_stream() {
 local -r image="$1" name="$2"
 local sha template

 zfs::checkenv
 sha=$(zfs::image_sha256 "${image}")
 template=$(zfs::ensure_template_from_stream "${image}" "${sha}")
 zfs::clone_container "${template}" "${name}"
}

# Exports a clone's @pristine snapshot as a zfs send stream file. Owns
# filename defaulting and the file-already-exists guard so runtime::export's
# ZFS branch is a single dispatch call.
zfs::export_to_file() {
 local -r name="$1"
 local filename="$2"

 if [ -z "${filename}" ]; then
 filename="${name}.zfs"
 fi
 filename=$(common::realpath "${filename}")
 if [ -e "${filename}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 common::err "File already exists: ${filename}"
 else
 rm -f "${filename}"
 fi
 fi

 common::log INFO "Creating zfs send stream..." NL
 zfs::send_stream "${name}" "${filename}"
}

# Materializes a template from a zfs send stream file. The cache key is the
# sha256 of the stream file (same scheme as the .sqsh path). Atomic via a
# .tmp dataset; integrates with the same eviction sweep as ensure_template.
zfs::ensure_template_from_stream() {
 local -r stream="$1" sha="$2"
 local -r store=$(zfs::store_dataset)
 local -r template="${store}/${zfs_template_subdir}/${sha}"
 local -r tmp="${template}.tmp"
 local -r snap="${template}@${zfs_pristine_snap}"
 local i timeout=600

 zfs::sweep_templates

 # Fast path: already cached.
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Ensure the templates parent exists before receive (zfs receive does not
 # auto-create parents).
 zfs create -p "${store}/${zfs_template_subdir}" 2> /dev/null || :

 if zfs receive -F "${tmp}" < "${stream}" 2> /dev/null; then
 # The received dataset brings its own snapshot. Rename the dataset to
 # the final template name; if the recv'd snapshot wasn't already named
 # @pristine, alias it.
 zfs rename "${tmp}" "${template}"
 if ! zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 local recvd_snap
 recvd_snap=$(zfs list -H -t snapshot -o name -r -d 1 "${template}" | head -1)
 [ -n "${recvd_snap}" ] && zfs rename "${recvd_snap}" "${snap}"
 fi
 zfs set readonly=on "${template}"
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Receive failed. Clean our orphan .tmp (if any) and wait for another
 # writer's @pristine.
 zfs destroy -r "${tmp}" 2> /dev/null || :
 for ((i = 0; i < timeout; i++)); do
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 printf "%s" "${template}"
 return
 fi
 sleep 1
 done
 common::err "Timed out waiting for stream receive: ${template}"
}

# Sends a clone's @pristine snapshot (or a fresh snapshot if the container is
# not a clone) to stdout. Used by --format=zfs export.
zfs::send_stream() {
 local -r name="$1" filename="$2"
 local -r store=$(zfs::store_dataset)
 local -r target="${store}/${name}"
 local origin

 if ! zfs list -H "${target}" > /dev/null 2>&1; then
 common::err "No such container: ${name}"
 fi

 origin=$(zfs get -H -o value origin "${target}")
 if [ -z "${origin}" ] || [ "${origin}" = "-" ]; then
 # Not a clone — must take a fresh snapshot of the live dataset.
 local snap="${target}@enroot-export-$$"
 zfs snapshot "${snap}"
 if ! zfs send "${snap}" > "${filename}"; then
 zfs destroy "${snap}" 2> /dev/null || :
 common::err "Failed to send stream for ${name}"
 fi
 zfs destroy "${snap}" 2> /dev/null || :
 else
 zfs send "${origin}" > "${filename}" \
 || common::err "Failed to send stream for ${name}"
 fi
}

# Materializes the merged Docker rootfs into a ZFS template (cached by
# cache_key) and clones it as the user's named container. Designed to be
# called from docker::load AFTER docker::_prepare_layers has populated the
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Plan C: .zfs image format (zfs send streams)#7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

sodre
merged 6 commits into

from

Apr 29, 2026
 Merged


Plan C: .zfs image format (zfs send streams)
#7
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
baf35ef

Add zfs::ensure_template_from_stream and zfs::send_stream helpers


sodre Apr 29, 2026
9f8d8fe

Dispatch runtime::create on .sqsh vs .zfs extension


sodre Apr 29, 2026
aeff6d0

Add --format=zfs to enroot export


sodre Apr 29, 2026
502531f

Mark Plan C (.zfs image format) as implemented


sodre Apr 29, 2026
ced1c56

Restore upstream comments dropped by earlier ZFS refactors


sodre Apr 29, 2026
4775c8d

Move Plan C ZFS-side helpers into storage_zfs.sh


sodre Apr 29, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
CLAUDE.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@ When debugging container behavior, the order is: image `/etc/{rc,fstab,environme
## Active design proposals

- **`doc/zfs.md`** — optional ZFS storage backend (`ENROOT_STORAGE_BACKEND=zfs`). Replaces `unsquashfs`-per-create with extract-once-then-`zfs clone`. Adds a `.zfs` (zfs send stream) image format and a `zfs://host/NAME` transport scheme alongside today's `.sqsh`. Introduces a shared template cache with a live/warm/cold lifecycle (knobs: `ENROOT_TEMPLATE_WARM_SECONDS`, `ENROOT_TEMPLATE_PRESSURE_THRESHOLD`; eviction is implicit on `create`, no daemon, no `enroot gc` command). Default backend (`dir`) is unchanged.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, Fmerged; B in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5)); C, D are still design-only.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, F, B merged; C in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5), [#7](https://github.com/zeroae/enroot/pull/7)); D is still design-only.

## Conventions

Expand Down

2 changes: 1 addition & 1 deletion
doc/zfs.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
# ZFS storage backend

This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. The remaining transports (`.zfs` file format, `zfs://` URI) are tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.
This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), C (`.zfs` image format), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. `enroot create` accepts both `.sqsh` and `.zfs` (zfs send stream) inputs; `enroot export --format=zfs` produces the latter. The remaining transport (`zfs://` URI) is tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.

## Motivation

Expand Down

17 changes: 14 additions & 3 deletions
enroot.in

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -170,8 +170,9 @@ enroot::usage() {
 Create a container image from a container root filesystem.

 Options:
 -o, --output Name of the output image file (defaults to "NAME.sqsh")
 -o, --output Name of the output image file (defaults to "NAME.sqsh" or "NAME.zfs")
 -f, --force Overwrite an existing container image
 --format Output format: "sqsh" (default) or "zfs" (zfs send stream; requires ZFS backend)
 EOF
 ;;
 import)
Expand DownExpand Up@@ -444,7 +445,7 @@ enroot::load() {
}

enroot::export() {
 local name= filename=
 local name= filename= format=sqsh

 while [ $# -gt 0 ]; do
 case "$1" in
Expand All@@ -462,6 +463,16 @@ enroot::export() {
 filename="${1#*=}"
 shift
 ;;
 --format)
 [ -z "${2-}" ] && enroot::usage export 1
 format="$2"
 shift 2
 ;;
 --format=*)
 [ -z "${1#*=}" ] && enroot::usage export 1
 format="${1#*=}"
 shift
 ;;
 -h|--help)
 enroot::usage export 0 ;;
 --)
Expand All@@ -477,7 +488,7 @@ enroot::export() {
 fi
 name="$1"

 runtime::export "${name}" "${filename}"
 runtime::export "${name}" "${filename}" "${format}"
}

enroot::create() {
Expand Down

93 changes: 68 additions & 25 deletions
src/runtime.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -437,23 +437,37 @@ runtime::create() {
 if [ ! -f "${image}" ]; then
 common::err "No such file or directory: ${image}"
 fi
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image}"
 fi

 # Resolve the container rootfs name.
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi

 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 case "${image}" in
 *.zfs)
 if ! zfs::enabled; then
 common::err ".zfs images require ENROOT_STORAGE_BACKEND=zfs"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.zfs}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 zfs::create_from_stream "${image}" "${rootfs}"
 ;;
 *)
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image} (expected .sqsh or .zfs)"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 ;;
 esac
}

runtime::_create_dir() {
Expand All@@ -462,6 +476,7 @@ runtime::_create_dir() {

 common::checkcmd unsquashfs find

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ -e "${rootfs}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
Expand All@@ -471,6 +486,7 @@ runtime::_create_dir() {
 fi
 fi

 # Extract the container rootfs from the image.
 common::log INFO "Extracting squashfs filesystem..." NL
 # XXX: https://github.com/NVIDIA/enroot/issues/90
 [ $(ulimit -n) -gt $((2**26)) ] && ulimit -n $((2**26))
Expand DownExpand Up@@ -549,19 +565,41 @@ runtime::load() {
}

runtime::export() {
 local rootfs="$1" filename="$2"
 local exclude=()

 common::checkcmd mksquashfs
 local rootfs_name="$1" filename="$2" format="${3:-sqsh}"

 # Resolve the container rootfs path.
 if [ -z "${rootfs}" ]; then
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 if [[ "${rootfs_name}" == */* ]]; then
 common::err "Invalid argument: ${rootfs_name}"
 fi

 case "${format}" in
 sqsh) ;;
 zfs)
 if ! zfs::enabled; then
 common::err "--format=zfs requires ENROOT_STORAGE_BACKEND=zfs"
 fi
 ;;
 *) common::err "Invalid format: ${format}" ;;
 esac

 if [ "${format}" = "sqsh" ]; then
 runtime::_export_sqsh "${rootfs_name}" "${filename}"
 else
 zfs::export_to_file "${rootfs_name}" "${filename}"
 fi
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs}")
}

runtime::_export_sqsh() {
 local -r rootfs_name="$1"
 local filename="$2"
 local rootfs exclude=()

 common::checkcmd mksquashfs

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi
Expand DownExpand Up@@ -657,6 +695,7 @@ runtime::list() {
runtime::remove() {
 local rootfs_name="$1"

 # Resolve the container rootfs path.
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
Expand All@@ -678,6 +717,8 @@ runtime::_remove_dir() {
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand All@@ -690,6 +731,8 @@ runtime::_remove_zfs() {
 local -r rootfs_name="$1"
 local rootfs
 rootfs="${ENROOT_DATA_PATH}/${rootfs_name}"

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand Down

118 changes: 118 additions & 0 deletions
src/storage_zfs.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -262,6 +262,124 @@ zfs::container_check() {
 fi
}

# Materializes a ZFS stream file into a template (cached by file sha) and
# clones it as the user's named container. Counterpart of zfs::ensure_template
# + zfs::clone_container for the .sqsh path; this is called from runtime::create
# when the input image has a .zfs extension.
zfs::create_from_stream() {
 local -r image="$1" name="$2"
 local sha template

 zfs::checkenv
 sha=$(zfs::image_sha256 "${image}")
 template=$(zfs::ensure_template_from_stream "${image}" "${sha}")
 zfs::clone_container "${template}" "${name}"
}

# Exports a clone's @pristine snapshot as a zfs send stream file. Owns
# filename defaulting and the file-already-exists guard so runtime::export's
# ZFS branch is a single dispatch call.
zfs::export_to_file() {
 local -r name="$1"
 local filename="$2"

 if [ -z "${filename}" ]; then
 filename="${name}.zfs"
 fi
 filename=$(common::realpath "${filename}")
 if [ -e "${filename}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 common::err "File already exists: ${filename}"
 else
 rm -f "${filename}"
 fi
 fi

 common::log INFO "Creating zfs send stream..." NL
 zfs::send_stream "${name}" "${filename}"
}

# Materializes a template from a zfs send stream file. The cache key is the
# sha256 of the stream file (same scheme as the .sqsh path). Atomic via a
# .tmp dataset; integrates with the same eviction sweep as ensure_template.
zfs::ensure_template_from_stream() {
 local -r stream="$1" sha="$2"
 local -r store=$(zfs::store_dataset)
 local -r template="${store}/${zfs_template_subdir}/${sha}"
 local -r tmp="${template}.tmp"
 local -r snap="${template}@${zfs_pristine_snap}"
 local i timeout=600

 zfs::sweep_templates

 # Fast path: already cached.
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Ensure the templates parent exists before receive (zfs receive does not
 # auto-create parents).
 zfs create -p "${store}/${zfs_template_subdir}" 2> /dev/null || :

 if zfs receive -F "${tmp}" < "${stream}" 2> /dev/null; then
 # The received dataset brings its own snapshot. Rename the dataset to
 # the final template name; if the recv'd snapshot wasn't already named
 # @pristine, alias it.
 zfs rename "${tmp}" "${template}"
 if ! zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 local recvd_snap
 recvd_snap=$(zfs list -H -t snapshot -o name -r -d 1 "${template}" | head -1)
 [ -n "${recvd_snap}" ] && zfs rename "${recvd_snap}" "${snap}"
 fi
 zfs set readonly=on "${template}"
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Receive failed. Clean our orphan .tmp (if any) and wait for another
 # writer's @pristine.
 zfs destroy -r "${tmp}" 2> /dev/null || :
 for ((i = 0; i < timeout; i++)); do
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 printf "%s" "${template}"
 return
 fi
 sleep 1
 done
 common::err "Timed out waiting for stream receive: ${template}"
}

# Sends a clone's @pristine snapshot (or a fresh snapshot if the container is
# not a clone) to stdout. Used by --format=zfs export.
zfs::send_stream() {
 local -r name="$1" filename="$2"
 local -r store=$(zfs::store_dataset)
 local -r target="${store}/${name}"
 local origin

 if ! zfs list -H "${target}" > /dev/null 2>&1; then
 common::err "No such container: ${name}"
 fi

 origin=$(zfs get -H -o value origin "${target}")
 if [ -z "${origin}" ] || [ "${origin}" = "-" ]; then
 # Not a clone — must take a fresh snapshot of the live dataset.
 local snap="${target}@enroot-export-$$"
 zfs snapshot "${snap}"
 if ! zfs send "${snap}" > "${filename}"; then
 zfs destroy "${snap}" 2> /dev/null || :
 common::err "Failed to send stream for ${name}"
 fi
 zfs destroy "${snap}" 2> /dev/null || :
 else
 zfs send "${origin}" > "${filename}" \
 || common::err "Failed to send stream for ${name}"
 fi
}

# Materializes the merged Docker rootfs into a ZFS template (cached by
# cache_key) and clones it as the user's named container. Designed to be
# called from docker::load AFTER docker::_prepare_layers has populated the
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Plan C: .zfs image format (zfs send streams)#7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

sodre
merged 6 commits into

from

Apr 29, 2026
 Merged


Plan C: .zfs image format (zfs send streams)
#7
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
baf35ef

Add zfs::ensure_template_from_stream and zfs::send_stream helpers


sodre Apr 29, 2026
9f8d8fe

Dispatch runtime::create on .sqsh vs .zfs extension


sodre Apr 29, 2026
aeff6d0

Add --format=zfs to enroot export


sodre Apr 29, 2026
502531f

Mark Plan C (.zfs image format) as implemented


sodre Apr 29, 2026
ced1c56

Restore upstream comments dropped by earlier ZFS refactors


sodre Apr 29, 2026
4775c8d

Move Plan C ZFS-side helpers into storage_zfs.sh


sodre Apr 29, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
CLAUDE.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@ When debugging container behavior, the order is: image `/etc/{rc,fstab,environme
## Active design proposals

- **`doc/zfs.md`** — optional ZFS storage backend (`ENROOT_STORAGE_BACKEND=zfs`). Replaces `unsquashfs`-per-create with extract-once-then-`zfs clone`. Adds a `.zfs` (zfs send stream) image format and a `zfs://host/NAME` transport scheme alongside today's `.sqsh`. Introduces a shared template cache with a live/warm/cold lifecycle (knobs: `ENROOT_TEMPLATE_WARM_SECONDS`, `ENROOT_TEMPLATE_PRESSURE_THRESHOLD`; eviction is implicit on `create`, no daemon, no `enroot gc` command). Default backend (`dir`) is unchanged.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, Fmerged; B in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5)); C, D are still design-only.
- **`doc/plans/`** — six implementation plans (A–F) breaking the ZFS backend into independently-landable slices. Start with `doc/plans/README.md` for the index and recommended landing order (A → E → F → B → C → D). Plans add a new sourced module `src/storage_zfs.sh` (under a `zfs::` namespace) and branch in `src/runtime.sh`, `src/docker.sh` on `ENROOT_STORAGE_BACKEND`. **Plans A, E, F, B merged; C in review** on `zenroot/main` (PRs [zeroae/enroot#1](https://github.com/zeroae/enroot/pull/1), [#2](https://github.com/zeroae/enroot/pull/2), [#3](https://github.com/zeroae/enroot/pull/3), [#5](https://github.com/zeroae/enroot/pull/5), [#7](https://github.com/zeroae/enroot/pull/7)); D is still design-only.

## Conventions

Expand Down

2 changes: 1 addition & 1 deletion
doc/zfs.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
# ZFS storage backend

This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. The remaining transports (`.zfs` file format, `zfs://` URI) are tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.
This document describes an optional ZFS-aware mode for the enroot container store. **Plans A (foundation), B (template warm/cold lifecycle), C (`.zfs` image format), E (ephemeral start), and F (Docker load) are implemented**: `enroot create`, `enroot remove`, ephemeral `enroot start <image>`, and `enroot load docker://...` all use ZFS datasets when `ENROOT_STORAGE_BACKEND=zfs`, with a shared template cache that survives `enroot remove` (warm) for `ENROOT_TEMPLATE_WARM_SECONDS` and gets pressure-evicted LRU once the templates dataset crosses `ENROOT_TEMPLATE_PRESSURE_THRESHOLD` of its quota. `enroot create` accepts both `.sqsh` and `.zfs` (zfs send stream) inputs; `enroot export --format=zfs` produces the latter. The remaining transport (`zfs://` URI) is tracked under `doc/plans/`. The default storage backend (plain directories under `ENROOT_DATA_PATH`) is unchanged and remains the only option on hosts without ZFS.

## Motivation

Expand Down

17 changes: 14 additions & 3 deletions
enroot.in

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -170,8 +170,9 @@ enroot::usage() {
 Create a container image from a container root filesystem.

 Options:
 -o, --output Name of the output image file (defaults to "NAME.sqsh")
 -o, --output Name of the output image file (defaults to "NAME.sqsh" or "NAME.zfs")
 -f, --force Overwrite an existing container image
 --format Output format: "sqsh" (default) or "zfs" (zfs send stream; requires ZFS backend)
 EOF
 ;;
 import)
Expand DownExpand Up@@ -444,7 +445,7 @@ enroot::load() {
}

enroot::export() {
 local name= filename=
 local name= filename= format=sqsh

 while [ $# -gt 0 ]; do
 case "$1" in
Expand All@@ -462,6 +463,16 @@ enroot::export() {
 filename="${1#*=}"
 shift
 ;;
 --format)
 [ -z "${2-}" ] && enroot::usage export 1
 format="$2"
 shift 2
 ;;
 --format=*)
 [ -z "${1#*=}" ] && enroot::usage export 1
 format="${1#*=}"
 shift
 ;;
 -h|--help)
 enroot::usage export 0 ;;
 --)
Expand All@@ -477,7 +488,7 @@ enroot::export() {
 fi
 name="$1"

 runtime::export "${name}" "${filename}"
 runtime::export "${name}" "${filename}" "${format}"
}

enroot::create() {
Expand Down

93 changes: 68 additions & 25 deletions
src/runtime.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -437,23 +437,37 @@ runtime::create() {
 if [ ! -f "${image}" ]; then
 common::err "No such file or directory: ${image}"
 fi
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image}"
 fi

 # Resolve the container rootfs name.
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi

 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 case "${image}" in
 *.zfs)
 if ! zfs::enabled; then
 common::err ".zfs images require ENROOT_STORAGE_BACKEND=zfs"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.zfs}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 zfs::create_from_stream "${image}" "${rootfs}"
 ;;
 *)
 if ! unsquashfs -s "${image}" > /dev/null 2>&1; then
 common::err "Invalid image format: ${image} (expected .sqsh or .zfs)"
 fi
 if [ -z "${rootfs}" ]; then
 rootfs=$(basename "${image%.sqsh}")
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 fi
 if zfs::enabled; then
 runtime::_create_zfs "${image}" "${rootfs}"
 else
 runtime::_create_dir "${image}" "${rootfs}"
 fi
 ;;
 esac
}

runtime::_create_dir() {
Expand All@@ -462,6 +476,7 @@ runtime::_create_dir() {

 common::checkcmd unsquashfs find

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ -e "${rootfs}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
Expand All@@ -471,6 +486,7 @@ runtime::_create_dir() {
 fi
 fi

 # Extract the container rootfs from the image.
 common::log INFO "Extracting squashfs filesystem..." NL
 # XXX: https://github.com/NVIDIA/enroot/issues/90
 [ $(ulimit -n) -gt $((2**26)) ] && ulimit -n $((2**26))
Expand DownExpand Up@@ -549,19 +565,41 @@ runtime::load() {
}

runtime::export() {
 local rootfs="$1" filename="$2"
 local exclude=()

 common::checkcmd mksquashfs
 local rootfs_name="$1" filename="$2" format="${3:-sqsh}"

 # Resolve the container rootfs path.
 if [ -z "${rootfs}" ]; then
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
 if [[ "${rootfs}" == */* ]]; then
 common::err "Invalid argument: ${rootfs}"
 if [[ "${rootfs_name}" == */* ]]; then
 common::err "Invalid argument: ${rootfs_name}"
 fi

 case "${format}" in
 sqsh) ;;
 zfs)
 if ! zfs::enabled; then
 common::err "--format=zfs requires ENROOT_STORAGE_BACKEND=zfs"
 fi
 ;;
 *) common::err "Invalid format: ${format}" ;;
 esac

 if [ "${format}" = "sqsh" ]; then
 runtime::_export_sqsh "${rootfs_name}" "${filename}"
 else
 zfs::export_to_file "${rootfs_name}" "${filename}"
 fi
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs}")
}

runtime::_export_sqsh() {
 local -r rootfs_name="$1"
 local filename="$2"
 local rootfs exclude=()

 common::checkcmd mksquashfs

 # Resolve the container rootfs path.
 rootfs=$(common::realpath "${ENROOT_DATA_PATH}/${rootfs_name}")
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi
Expand DownExpand Up@@ -657,6 +695,7 @@ runtime::list() {
runtime::remove() {
 local rootfs_name="$1"

 # Resolve the container rootfs path.
 if [ -z "${rootfs_name}" ]; then
 common::err "Invalid argument"
 fi
Expand All@@ -678,6 +717,8 @@ runtime::_remove_dir() {
 if [ ! -d "${rootfs}" ]; then
 common::err "No such file or directory: ${rootfs}"
 fi

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand All@@ -690,6 +731,8 @@ runtime::_remove_zfs() {
 local -r rootfs_name="$1"
 local rootfs
 rootfs="${ENROOT_DATA_PATH}/${rootfs_name}"

 # Remove the rootfs specified after asking for confirmation.
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 read -r -e -p "Do you really want to delete ${rootfs}? [y/N] "
 fi
Expand Down

118 changes: 118 additions & 0 deletions
src/storage_zfs.sh

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -262,6 +262,124 @@ zfs::container_check() {
 fi
}

# Materializes a ZFS stream file into a template (cached by file sha) and
# clones it as the user's named container. Counterpart of zfs::ensure_template
# + zfs::clone_container for the .sqsh path; this is called from runtime::create
# when the input image has a .zfs extension.
zfs::create_from_stream() {
 local -r image="$1" name="$2"
 local sha template

 zfs::checkenv
 sha=$(zfs::image_sha256 "${image}")
 template=$(zfs::ensure_template_from_stream "${image}" "${sha}")
 zfs::clone_container "${template}" "${name}"
}

# Exports a clone's @pristine snapshot as a zfs send stream file. Owns
# filename defaulting and the file-already-exists guard so runtime::export's
# ZFS branch is a single dispatch call.
zfs::export_to_file() {
 local -r name="$1"
 local filename="$2"

 if [ -z "${filename}" ]; then
 filename="${name}.zfs"
 fi
 filename=$(common::realpath "${filename}")
 if [ -e "${filename}" ]; then
 if [ -z "${ENROOT_FORCE_OVERRIDE-}" ]; then
 common::err "File already exists: ${filename}"
 else
 rm -f "${filename}"
 fi
 fi

 common::log INFO "Creating zfs send stream..." NL
 zfs::send_stream "${name}" "${filename}"
}

# Materializes a template from a zfs send stream file. The cache key is the
# sha256 of the stream file (same scheme as the .sqsh path). Atomic via a
# .tmp dataset; integrates with the same eviction sweep as ensure_template.
zfs::ensure_template_from_stream() {
 local -r stream="$1" sha="$2"
 local -r store=$(zfs::store_dataset)
 local -r template="${store}/${zfs_template_subdir}/${sha}"
 local -r tmp="${template}.tmp"
 local -r snap="${template}@${zfs_pristine_snap}"
 local i timeout=600

 zfs::sweep_templates

 # Fast path: already cached.
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Ensure the templates parent exists before receive (zfs receive does not
 # auto-create parents).
 zfs create -p "${store}/${zfs_template_subdir}" 2> /dev/null || :

 if zfs receive -F "${tmp}" < "${stream}" 2> /dev/null; then
 # The received dataset brings its own snapshot. Rename the dataset to
 # the final template name; if the recv'd snapshot wasn't already named
 # @pristine, alias it.
 zfs rename "${tmp}" "${template}"
 if ! zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 local recvd_snap
 recvd_snap=$(zfs list -H -t snapshot -o name -r -d 1 "${template}" | head -1)
 [ -n "${recvd_snap}" ] && zfs rename "${recvd_snap}" "${snap}"
 fi
 zfs set readonly=on "${template}"
 zfs::touch_template "${template}"
 printf "%s" "${template}"
 return
 fi

 # Receive failed. Clean our orphan .tmp (if any) and wait for another
 # writer's @pristine.
 zfs destroy -r "${tmp}" 2> /dev/null || :
 for ((i = 0; i < timeout; i++)); do
 if zfs list -H -t snapshot "${snap}" > /dev/null 2>&1; then
 printf "%s" "${template}"
 return
 fi
 sleep 1
 done
 common::err "Timed out waiting for stream receive: ${template}"
}

# Sends a clone's @pristine snapshot (or a fresh snapshot if the container is
# not a clone) to stdout. Used by --format=zfs export.
zfs::send_stream() {
 local -r name="$1" filename="$2"
 local -r store=$(zfs::store_dataset)
 local -r target="${store}/${name}"
 local origin

 if ! zfs list -H "${target}" > /dev/null 2>&1; then
 common::err "No such container: ${name}"
 fi

 origin=$(zfs get -H -o value origin "${target}")
 if [ -z "${origin}" ] || [ "${origin}" = "-" ]; then
 # Not a clone — must take a fresh snapshot of the live dataset.
 local snap="${target}@enroot-export-$$"
 zfs snapshot "${snap}"
 if ! zfs send "${snap}" > "${filename}"; then
 zfs destroy "${snap}" 2> /dev/null || :
 common::err "Failed to send stream for ${name}"
 fi
 zfs destroy "${snap}" 2> /dev/null || :
 else
 zfs send "${origin}" > "${filename}" \
 || common::err "Failed to send stream for ${name}"
 fi
}

# Materializes the merged Docker rootfs into a ZFS template (cached by
# cache_key) and clones it as the user's named container. Designed to be
# called from docker::load AFTER docker::_prepare_layers has populated the
Expand Down