ZFS backend: support pyxis (Slurm) workflows where enroot runs after privilege-drop to the job user

[sodre]

User story

As an HPC operator running NVIDIA pyxis on Slurm with the zenroot ZFS backend, I want pyxis-launched containers (srun --container-image=...) to use the ZFS backend's clone-on-create path, so that container startup is near-instant (zfs clone) instead of going through a per-job mksquashfs rebuild.

Today the ZFS backend works perfectly when invoked as root, but breaks for the pyxis path because pyxis correctly drops to the user's UID before invoking enroot create, and unprivileged enroot create blocks for ten minutes on template extraction (the documented mount(2) limitation in doc/zfs.md).

Problem statement

doc/zfs.md flags this:

Linux mount(2) limitation: Unprivileged users cannot call mount(2) even with ZFS delegation, so datasets won't auto-mount. Solutions include running enroot create as root or using user namespaces with fs.namespace.unprivileged_userns_clone=1.

In the documented Slurm + pyxis topology this isn't a workaround anyone can apply at the calling layer:

• slurmstepd starts as root, then drops to the job user (uid 1000 in the repro below) before invoking enroot import / enroot create via the pyxis SPANK plugin.
• That is pyxis's correct security posture; container processes run as the submitter, not root.
• kernel.unprivileged_userns_clone=1 is already set on the host, but enroot create still hangs ten minutes on Timed out waiting for template extraction.
• The result: pyxis fails with error: Failed to invoke spank plugin stack, ten minutes after job launch, on every container job.

This is a real-world blocker on a documented integration target (the upstream NVIDIA/enroot README points at pyxis as the canonical Slurm story).

What should work

After whatever changes are needed, the following should succeed end-to-end on a node where the ZFS backend is configured per doc/zfs.md and pyxis is installed normally:

# Submitter is unprivileged (uid 1000); slurmstepd starts root and drops.
srun --gres=gpu:1 --container-image=ubuntu:24.04 cat /etc/os-release

Acceptance criteria:

1. The above completes in seconds on first run (template extracted), and near-instant on subsequent runs (clone-on-create benefit visible to the operator).
2. Direct enroot create <foo>.sqsh run as the unprivileged submitter (no sudo) outside Slurm also succeeds.
3. The unprivileged path does not require operator-managed wrappers, suid binaries, or AppArmor profile churn beyond what doc/zfs.md prescribes.
4. Documentation in doc/zfs.md covers the Slurm + pyxis setup explicitly: any required sysctls, dataset properties, ZFS delegations, and (if applicable) the user-namespace plumbing that makes the unprivileged path work.

How the maintainer satisfies these (user-namespace mount delegation, suid enroot-mount helper, an opt-in "pre-mount on the operator side" path, etc.) is out of scope for this issue.

Repro environment

• 3-node cluster: spark-ctrl (Pi 5, Debian 13) runs slurmctld + slurmdbd + MariaDB; two DGX Sparks (spark-f2ff, spark-4c55, Ubuntu 24.04, GB10 / arm64) run slurmd.
• Slurm 25.11.5, built from source; auth/slurm + cred/slurm; SlurmUser=slurm uniform UID 64030.
• pyxis v0.23.0, built against the locally-installed Slurm headers; loaded via plugstack.conf:

  required /usr/local/lib/slurm/spank_pyxis.so runtime_path=/var/lib/pyxis-runtime
  

• enroot packages: enroot_4.1.2.zfs.1-1_arm64.deb and enroot+caps_4.1.2.zfs.1-1_arm64.deb (this fork, latest release).
• ZFS layout per doc/zfs.md:

  tank/enroot/data mountpoint=/var/lib/enroot
  tank/enroot/data/.templates # auto-created by enroot
  

• ZFS delegation:

  zfs allow sodre create,mount,clone,destroy,snapshot,rename,hold,release tank/enroot/data
  zfs allow sodre receive tank/enroot/data/.templates
  

• /etc/enroot/enroot.conf:

  ENROOT_STORAGE_BACKEND zfs
  ENROOT_CACHE_PATH /var/cache/enroot
  ENROOT_DATA_PATH /var/lib/enroot
  ENROOT_TEMP_PATH /var/tmp/enroot
  ENROOT_RUNTIME_PATH /run/enroot
  ENROOT_SQUASH_OPTIONS -comp lzo -noD -processors 20
  

• Sysctl: kernel.unprivileged_userns_clone = 1.
• subuid/subgid: sodre:100000:65536.

Reproducer

Works (as root)

$ sudo enroot import -o /tmp/hello.sqsh docker://hello-world # ~0.6 s
$ sudo enroot create --name test-hw /tmp/hello.sqsh # ~0.3 s
$ sudo enroot create --name test-hw2 /tmp/hello.sqsh # ~0.3 s — clone-on-create
$ sudo enroot list
test-hw
test-hw2
$ sudo zfs list -r tank/enroot/data
NAME USED AVAIL REFER
tank/enroot/data 348K 2.65T 104K
tank/enroot/data/.templates 236K 2.65T 128K
tank/enroot/data/.templates/812f6e818527af65ea91417ba89e1ffe4a5b73409169fced5c073b636f278fac 108K 2.65T 108K
tank/enroot/data/test-hw 8K 2.65T 108K
tank/enroot/data/test-hw2 8K 2.65T 108K

Hangs (as the unprivileged submitter)

$ enroot import -o /tmp/hello.sqsh docker://hello-world # OK
$ time enroot create --name test-hw /tmp/hello.sqsh
[ERROR] Timed out waiting for template extraction:
 tank/enroot/data/.templates/<sha>.
 A previous extractor may have crashed; remove tank/enroot/data/.templates/<sha>.tmp
 manually and retry.

real 10m2.759s

No .tmp file is left behind. The dataset tank/enroot/data/.templates/<sha> is created but appears empty / unmounted from the user's perspective. ZFS perms include mount, but mount(2) is rejected by the kernel (unrelated to ZFS delegation).

Hangs the same way under pyxis

$ srun --gres=gpu:1 --container-image=ubuntu:24.04 cat /etc/os-release
[2026-04-29T13:28:16] error: Failed to invoke spank plugin stack
srun: error: spark-4c55: task 0: Exited with exit code 1

real 10m11.650s

slurmstepd's enroot create invocation runs as uid 1000 (post-privilege-drop) and hits the same template-extraction timeout.

Out of scope for this ticket

• Choice of solution. User namespaces, a small suid helper, an operator-side template-warm daemon, an opt-in flag that pre-mounts as root then chowns — all reasonable; deciding between them is a design question for this fork.
• Whether to keep the dir backend as the default (it should — see doc/zfs.md's explicit "opt-in" framing).
• pyxis-side configuration. This is purely about making enroot's ZFS backend usable from an unprivileged caller in the standard way pyxis invokes it.