Report exhausted host fds as ENOMEM, not EBADF

[xalestar]

Summary

ppoll and pselect6 duplicate every descriptor in a wait list while sibling threads are alive. Running the host descriptor table out that way reached the guest as POLLNVAL or EBADF on descriptors that were still open, and the standard reaction to both is to close them. Exhaustion returns ENOMEM now, which is what Linux reports for a wait whose tables it cannot build.

Follow-up to #316 and #318, which covered descriptors the host poll() refuses. This one is about descriptors the host never got to look at.

Measured

Host soft limit 1280, one temp file duped until EMFILE. Single-threaded guests borrow rather than duplicate, so the second thread is what triggers it.

 main (628181b) this branch
ppoll(nfds=256) ret=256 POLLNVAL=4 ret=-1 ENOMEM POLLNVAL=0
pselect(nfds=1024) ret=-1 EBADF ret=-1 ENOMEM
still open afterwards 256/256, 1021/1021 unchanged

Scope notes

The other callers inherit the split.host_fd_ref_open_io{,_gen,_state} has 24 call sites, 22 outside the wait paths, and all but fuse.c propagate the return value. read, ioctl, splice, copy_file_range, fsync and friends answer an exhausted host table with ENOMEM now. They were returning EBADF for a descriptor that is open, so the path was already wrong; ENOMEM is undocumented for a couple of them (read, lseek) and documented for others (splice, copy_file_range, process_vm_readv). No existing test depended on the old value.

host_fd_ref_open is left alone. The 0/-1 spelling next door collapses the same two failures across 54 call sites that each pick their own errno. That is a return-convention change, not an added case.

Descriptor usage is unchanged. A wait consumes threads x list length, which HOST_FD_RESERVE's two-per-thread budget does not cover, as the comment in src/elfuse-limits.h notes. The call still fails; it just stops blaming the caller's descriptors.

Test plan

tests/test-poll-fd-exhaustion.c holds a sibling thread open, fills the descriptor table, and requires that no still-open descriptor comes back POLLNVAL and that pselect6 does not fail with EBADF. In tests/manifest.txt with host_nofile=elfuse-minimum, and in tests/test-matrix.sh.

Check	Result
new test vs main (628181b)	2 failed
new test vs this branch	2 passed
new test vs real kernel, qemu-system-aarch64	2 passed
make check, make check-format, all six .ci/*.sh	rc=0
build/test-poll	22 passed

The reference-kernel run is what backs the matrix entry: a real kernel never duplicates here, so the test asserts Linux behaviour rather than an elfuse-specific expectation.

---

Summary by cubic

Report host FD table exhaustion as ENOMEM instead of EBADF/POLLNVAL. Previously, ppoll marked entries POLLNVAL and pselect6 failed with EBADF when duplication ran out; now both return ENOMEM, preventing valid descriptors from being closed and matching Linux.

Notes for review

• Split failures in host_fd_ref_open_io_state: return ENOMEM when dup() fails after a live snapshot; return EBADF for closed, out-of-range, or O_PATH descriptors.
• Update ppoll/pselect6 to propagate ENOMEM on mid-list exhaustion and avoid emitting POLLNVAL/EBADF for open descriptors; other callers of host_fd_ref_open_io{,_gen,_state} inherit ENOMEM on host table exhaustion, while host_fd_ref_open is unchanged.
• Add tests/test-poll-fd-exhaustion.c and wire it into the build/matrix to verify no POLLNVAL/EBADF on open descriptors and ENOMEM on exhaustion.

^{Written for commit 7b738af. Summary will update on new commits.}

[cubic-dev-ai]

All reported issues were addressed across 6 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>