Uh oh!
There was an error while loading. Please reload this page.
fs: read small files in one thread pool round trip - #65327
Conversation
0eaf058 to
21db0abCompareCodecov Reportβ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@## main #65327 +/- ##
==========================================
- Coverage 90.13% 90.10% -0.04%
==========================================
Files 752 752 Lines 251568 251915 +347 Branches 47270 47353 +83 ==========================================
+ Hits 226759 226976 +217 - Misses 16168 16266 +98 - Partials 8641 8673 +32
π New features to boost your workflow:
|
nodejs-github-bot
commented
Aug 16, 2026
Uh oh!
There was an error while loading. Please reload this page.
jasnell
left a comment
There was a problem hiding this comment.
Test failure on AIX needs to be looked at. Also, are these reads actually abortable in any way?
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
fs.readFile(path) took four libuv thread pool round trips for a typical small file -- open, fstat, read and close, each its own uv_fs request with its own queue wait, completion callback and JS/C++ crossing -- and fs.promises.readFile(path) did the same through a FileHandle. For the small files applications read most, the round trips are the cost, and each occupies a slot in the pool that concurrent dns.lookup(), fs and crypto work is also queueing for. Add ReadFileJob (an AsyncWrap + ThreadPoolWork) that performs open + fstat + read-to-EOF + close as one pool task and reports the whole content, or, when the file turns out to be larger than one chunk (kReadFileBufferLength, 512 KiB), stops after fstat() and hands the fd and size back so that the existing chunked reader continues unchanged (large reads stay interleaved and abortable between chunks, and still save the fstat round trip). fs.readFile() and fs.promises.readFile() use it for path arguments without a user buffer; file descriptors, FileHandles, options.buffer and an active VFS keep their paths. Behavior is otherwise kept: same bytes for every size and encoding; open failures report syscall 'open' with the path, read failures 'read'; permission errors are delivered through the callback/promise as before; an abort that arrives while the read is in flight still wins; the job is an FSREQCALLBACK resource for async_hooks; a handed back fd is tracked exactly like one from a plain open(). Tests that asserted the internal open/fstat/read/close request chain, used readFile() as a proxy for an fstat trace event, or injected faults through FileHandle.prototype for path-based reads are adjusted to keep testing what they test (a file just over one chunk where the chain shape matters, fs.fstat() for the fstat trace, a larger file so the FileHandle path is taken). fs.readFile() of 4 KiB files at concurrency 64 goes from ~51k to ~306k files per second, and a mixed stat/readFile/dns.lookup burst from ~66k to ~312k operations per second. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
21db0ab to
b626e93Comparecodebytere
commented
Aug 17, 2026
@jasnell re aborts: the single round trip itself isn't interruptible once it's on the pool, same as an individual read req now; an already-aborted signal never schedules it, an abort that lands while it's in flight wins when it completes, and anything over one chunk hands the fd back to the existing chunked reader, so those stay abortable between chunks exactly as before. |
nodejs-github-bot
commented
Aug 17, 2026
nodejs-github-bot
commented
Aug 19, 2026
Uh oh!
There was an error while loading. Please reload this page.
nodejs-github-bot
commented
Aug 19, 2026
Landed in 542e2b2 |
fs.readFile(path) took four libuv thread pool round trips for a typical small file -- open, fstat, read and close, each its own uv_fs request with its own queue wait, completion callback and JS/C++ crossing -- and fs.promises.readFile(path) did the same through a FileHandle. For the small files applications read most, the round trips are the cost, and each occupies a slot in the pool that concurrent dns.lookup(), fs and crypto work is also queueing for. Add ReadFileJob (an AsyncWrap + ThreadPoolWork) that performs open + fstat + read-to-EOF + close as one pool task and reports the whole content, or, when the file turns out to be larger than one chunk (kReadFileBufferLength, 512 KiB), stops after fstat() and hands the fd and size back so that the existing chunked reader continues unchanged (large reads stay interleaved and abortable between chunks, and still save the fstat round trip). fs.readFile() and fs.promises.readFile() use it for path arguments without a user buffer; file descriptors, FileHandles, options.buffer and an active VFS keep their paths. Behavior is otherwise kept: same bytes for every size and encoding; open failures report syscall 'open' with the path, read failures 'read'; permission errors are delivered through the callback/promise as before; an abort that arrives while the read is in flight still wins; the job is an FSREQCALLBACK resource for async_hooks; a handed back fd is tracked exactly like one from a plain open(). Tests that asserted the internal open/fstat/read/close request chain, used readFile() as a proxy for an fstat trace event, or injected faults through FileHandle.prototype for path-based reads are adjusted to keep testing what they test (a file just over one chunk where the chain shape matters, fs.fstat() for the fstat trace, a larger file so the FileHandle path is taken). fs.readFile() of 4 KiB files at concurrency 64 goes from ~51k to ~306k files per second, and a mixed stat/readFile/dns.lookup burst from ~66k to ~312k operations per second. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com> PR-URL: #65327 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: James M Snell <jasnell@gmail.com>
fs.readFile(path) took four libuv thread pool round trips for a typical small file -- open, fstat, read and close, each its own uv_fs request with its own queue wait, completion callback and JS/C++ crossing -- and fs.promises.readFile(path) did the same through a FileHandle. For the small files applications read most, the round trips are the cost, and each occupies a slot in the pool that concurrent dns.lookup(), fs and crypto work is also queueing for. Add ReadFileJob (an AsyncWrap + ThreadPoolWork) that performs open + fstat + read-to-EOF + close as one pool task and reports the whole content, or, when the file turns out to be larger than one chunk (kReadFileBufferLength, 512 KiB), stops after fstat() and hands the fd and size back so that the existing chunked reader continues unchanged (large reads stay interleaved and abortable between chunks, and still save the fstat round trip). fs.readFile() and fs.promises.readFile() use it for path arguments without a user buffer; file descriptors, FileHandles, options.buffer and an active VFS keep their paths. Behavior is otherwise kept: same bytes for every size and encoding; open failures report syscall 'open' with the path, read failures 'read'; permission errors are delivered through the callback/promise as before; an abort that arrives while the read is in flight still wins; the job is an FSREQCALLBACK resource for async_hooks; a handed back fd is tracked exactly like one from a plain open(). Tests that asserted the internal open/fstat/read/close request chain, used readFile() as a proxy for an fstat trace event, or injected faults through FileHandle.prototype for path-based reads are adjusted to keep testing what they test (a file just over one chunk where the chain shape matters, fs.fstat() for the fstat trace, a larger file so the FileHandle path is taken). fs.readFile() of 4 KiB files at concurrency 64 goes from ~51k to ~306k files per second, and a mixed stat/readFile/dns.lookup burst from ~66k to ~312k operations per second. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com> PR-URL: #65327 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: James M Snell <jasnell@gmail.com>
mock-fs throws at require time on Node 26.8+ (nodejs/node#65327 made fs.readFile bypass binding.open, breaking its ReadFileContext sniffing; tschaub/mock-fs#447). The test only needed empty files on disk for glob threshold matching, so resolve the relative and glob thresholds against a checked-in fixture tree instead and drop the dependency.
Node 26.8.0 rewrote fs.readFile (nodejs/node#65327) to do open, fstat, read and close in a single thread pool round trip, so binding.open is no longer called from JS. mock-fs recovers the ReadFileContext prototype by intercepting binding.open during a dummy readFile, so it now gets undefined and throws at require time: TypeError: Cannot read properties of undefined (reading 'read') at exports.patchReadFileContext (mock-fs/lib/readfilecontext.js:40:30) Because it throws on require rather than in a test, it takes out config_test.ts and file_auth_test.ts in full. The matrix entries are floating majors, so setup-node resolves '26' to whatever the newest 26.x is at run time. That is why main went red on the merge of kubernetes-client#3022 without any change to the code under test: the branch last ran CI on 26.7.0, and by the time it merged five days later the runner had picked up 26.8.1. Pinning to 26.7 restores a green build. It is a stopgap: mock-fs has had no functional release since February 2025 and the upstream report (tschaub/mock-fs#447) is unanswered, so the durable fix is to stop depending on it.
fs.writeFile(path, data) took three libuv thread pool round trips (open, write, close), each its own request with its own queue wait, completion callback and JS/C++ crossing, and fs.promises.writeFile() did the same through a FileHandle. For the small files applications write most, the round trips are the cost, and each occupies a pool slot that concurrent fs, dns.lookup() and crypto work is also queueing for. Add WriteFileJob next to ReadFileJob: an AsyncWrap + ThreadPoolWork that opens, writes the whole buffer (looping on short writes) and closes as one pool task, keeping the buffer alive until it is done. fs.writeFile() uses it for path arguments without flush; fs.promises.writeFile() additionally keeps data above one write chunk (and iterables) on the FileHandle path, so large writes stay abortable between chunks as before. File descriptors, FileHandles, flush: true and an active VFS keep their existing paths. Behavior is otherwise kept: open failures report syscall 'open' with the path, write failures 'write'; permission errors are delivered through the callback/promise; an abort signalled while the write is in flight is still reported as an AbortError; the job is an FSREQCALLBACK resource for async_hooks and emits the 'write' fs trace event. Tests that used fs.writeFile() as a proxy for open/close trace events, or injected FileHandle faults for path-based writes, are adjusted to keep testing what they test. The job holds the buffer's backing store, so the memory stays valid if the buffer is detached or collected before the write finishes; a resizable ArrayBuffer could still have its pages decommitted by a shrink, so its contents are copied when the job is created. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com> PR-URL: #65489 Refs: #65327 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>
fs.readFile()/fs.promises.readFile()of small files get 3β6Γ faster, and use one libuv thread-pool task instead of four, by doing open + fstat + read + close in a single round trip.(Linux x64,
--set duration=2, 30 runs.)Today a path-based
readFileissuesopen,fstat,readandcloseas four separateuv_fs_*requests, each with its own queue wait, completion callback and JSβC++ crossing; the promise API does the same through aFileHandle. For small files those round trips are the whole cost, and each one takes a pool slot away from concurrent dns/zlib/crypto/fs work.ReadFileJob(anAsyncWrap+ThreadPoolWork, providerFSREQCALLBACK) runs open + fstat + read-to-EOF + close as one task and returns the content. If the file is larger than one chunk (kReadFileBufferLength, 512 KiB) it stops afterfstatand hands back the fd and size, and the existing chunked reader continues exactly as today (interleaved, abortable between chunks). BothreadFiles use it for path arguments without a user buffer; fds andFileHandles are unchanged.Preserved on purpose: identical results for every size/encoding;
openerrors reportsyscall: 'open'+path, read errors'read'; permission errors arrive through the callback/promise; an abort that lands while the round trip is in flight still wins; the handed-back fd is tracked and closed like any other; size-0 files (procfs) are read to EOF.One open point: 16β32 MiB reads via
fs.promises.readFile(β¦, 'utf-8')at concurrency 10 measure β2β¦3 % (***), reproducibly; the same sizes as Buffers, via the callback API, or at concurrency 1 are flat. They take the hand-back path with identical syscalls, and direct timing shows β€2 %, so I haven't pinned it down. If preferred, the promise API can keep its current path and only the callback API changes.Tests:
test-fs-readfile-one-roundtrip.js(new; also passes on currentmain): sizes across the 512 KiB threshold, encodings, flags, error shapes, abort before/during, fd/FileHandleinputs, no fd leak on hand-back, procfs/sysfs, async_hooks lifecycle. Adjusted to keep testing what they test:test/async-hooks/test-fsreqcallback-readFile.js(accepts one request instead of exactly four),test-graph.fsreq-readFile.js(reads a 512 KiB+1 file so the four-request chain keeps its shape),test-async-exec-resource-match.js(resource + β₯1 fs request),test-trace-events-fs-async.js(usesfs.fstat()for the fstat trace instead ofreadFileas a proxy),test-fs-promises-readfile.js(zero-size-liar case goes through aFileHandle),test-fs-promises-file-handle-{op,aggregate,close}-errors.js(use a >512 KiB file so the patchedFileHandlepath is taken). fs, async-hooks, permission, worker, process and child_process suites pass.Disclosure: the code, test, measurements and this description were written by Claude Code, directed and reviewed by @codebytere.