runtime page size detection + rework GeneralPurposeAllocator to reduce active mapping count + Allocator VTable API update

[archbirdplus]

release notes

---

This PR implements runtime page sizes. Closes#4082. Closes#11308. Closes#16331. Closes#20038. Closes#20714.

Notable changes:

• Remove mem.page_size.
• Add heap.min_page_size, heap.max_page_size, heap.pageSize().
• Add options.min_page_size, options.max_page_size, options.queryPageSizeFn.

Not implemented:

• A special-cased align_page alignment value. Use align(std.heap.min_page_size) on your pointers instead.

[cn-cn-cn]

Hi, thanks so much for putting in the work to make runtime page size a reality :) These else prongs in heap.zig worry me, it seems too easy to add an architecture and have an incorrect page size set as default. I think a compile error would be more appropriate. Also, I think it would be good if pageSize was marked inline to propagate the comptime-ness of the result.

[archbirdplus]

@notcancername

Inlining:
Looks like pageSize is already comptime and inlined during optimization. Comptime is a problem though. It suggests to some people that their code is valid when it isn't. I would like to ban the use of this function in comptime-dependent cases, demanding that the user prove that pageSize will be comptime (in which case they should use page_size instead). But sniffing for comptime is a bug, not a feature: #7539 (comment).
Since inlining has no semantic difference, I'll inline it anyways.

Else:
Do you mean the page_size else prongs? I suppose it's fine to break the few projects that depended on that.

[cn-cn-cn]

I would like to ban the use of this function in comptime-dependent cases, demanding that the user prove that pageSize will be comptime (in which case they should use page_size instead).

Thanks, I am in favor of this. I believe you may be able to achieve this by assigning to a var and returning that. For what it's worth, #868 has been implemented, so you might want to use that.

Do you mean the page_size else prongs? I suppose it's fine to break the few projects that depended on that.

Yes, page_size should match all current CPU architectures, so that when a new architecture is added, the implementer gets a compile error when code tries to access page_size (ditto for page_size_cap), and adds the correct value.

[archbirdplus]

Local tests now pass. I just can't figure out how to test on Windows.

[archbirdplus]

I'd like to re-run CI. I cannot get local tests to pass, but I would like to see how the CI runner behaves.

There seems to be another issue besides page size that prevents me from running tests myself.
Running the test command from the CI script on Asahi Linux fails like so:

+ ./stage4/bin/zig build test docs --maxrss 24696061952 -fwasmtime -Denable-llvm -Dtarget=native-native-musl --zig-lib-dir /home/arch/zig/lib --cache-dir /home/arch/zig/builddir-alt/.zig-cache -Dskip-release
test
└─ zig build-exe zig Debug native-native-musl 2 errors
error: ld.lld: undefined symbol: __libc_single_threaded
note: referenced by zig_llvm.cpp
note: zig_llvm.cpp.o:(ZigLLVMCreateTargetMachine) in archive /home/arch/zig/builddir-alt/zigcpp/libzigcpp.a
note: referenced by zig_llvm.cpp
note: zig_llvm.cpp.o:(ZigLLVMCreateTargetMachine) in archive /home/arch/zig/builddir-alt/zigcpp/libzigcpp.a
note: referenced by zig_clang.cpp
note: zig_clang.cpp.o:(ZigClangLoadFromCommandLine) in archive /home/arch/zig/builddir-alt/zigcpp/libzigcpp.a
note: referenced 17 more times
error: ld.lld: undefined symbol: __isoc23_strtol
note: referenced by zig_llvm.cpp
note: zig_llvm.cpp.o:(ZigLLVMTargetMachineEmitToFile) in archive /home/arch/zig/builddir-alt/zigcpp/libzigcpp.a
error: the following command failed with 2 compilation errors:
/home/arch/zig/builddir-alt/stage4/bin/zig build-exe --stack 33554432 /home/arch/zig/builddir-alt/zigcpp/libzigcpp.a /usr/lib64/libclang-cpp.so /usr/lib64/liblldMinGW.so /usr/lib64/liblldELF.so /usr/lib64/liblldCOFF.so /usr/lib64/liblldWasm.so /usr/lib64/liblldMachO.so /usr/lib64/liblldCommon.so /usr/lib64/libLLVM-18.so -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=600 -ltinfo /usr/lib64/libz.so /usr/lib/gcc/aarch64-redhat-linux/14/libstdc++.so -lunwind -fno-sanitize-thread -ODebug -target native-native-musl -mcpu native -I /usr/include -I /usr/include -L /usr/lib64 --dep aro --dep aro_translate_c --dep build_options -Mroot=/home/arch/zig/src/main.zig -Maro=/home/arch/zig/lib/compiler/aro/aro.zig --dep aro -Maro_translate_c=/home/arch/zig/lib/compiler/aro_translate_c.zig -Mbuild_options=/home/arch/zig/builddir-alt/.zig-cache/c/e867395cb58fc12949748f95b1b06d98/options.zig -lc --cache-dir /home/arch/zig/builddir-alt/.zig-cache --global-cache-dir /home/arch/.cache/zig --name zig --zig-lib-dir /home/arch/zig/lib/ --listen=-
test
└─ test-std
└─ run test std-wasm32-wasi.0.1...0.1-musl-generic-Debug-libc 2600/2718 passed, 1 failed, 117 skipped
error: 'fs.test.test.file operations on directories' failed: expected error.IsDir, found fs.File{ .handle = 7 }
Unable to dump stack trace: not implemented for Wasm
error: while executing test 'zig.system.darwin.macos.test.detect', the following test command failed:
wasmtime --dir=. -- /home/arch/zig/builddir-alt/.zig-cache/o/0d427260497117aec1db32d42886460a/test.wasm --seed=0x563739fd --listen=-

Whether I use this PR or just edit the page_size alone, the errors are the same, so at least I haven't created any new errors. Before rebasing this PR today, I had completely different errors about the wasmtime test runner reaching unreachable code while parsing arguments.
Running tests without any changes to Zig would crash with the classic #16331.

Do these errors suggest anything in particular? Maybe I didn't replicate the CI script closely enough?

[alexrp]

I think you might need to rebase this on master for CI to run? 🤔

[archbirdplus]

Alright, rebased again. Though what I understood was that CI needed to be invoked by a Zig member. Hopefully I can fix up the stylistic issues soon.

[rootbeer]

For the wasm failure in your local run of the tests, I see:

error: 'fs.test.test.file operations on directories' failed: expected error.IsDir, found fs.File{ .handle = 7 }
Unable to dump stack trace: not implemented for Wasm
error: while executing test 'zig.system.darwin.macos.test.detect', the following test command failed:
wasmtime --dir=. -- /home/arch/zig/builddir-alt/.zig-cache/o/0d427260497117aec1db32d42886460a/test.wasm --seed=0x563739fd --listen=-

I suspect this is from using a too-recent wasmtime, and downgrading to v10.0.2 (which is the version CI runs) will fix it. See #20747.

[linusg]

This broke the UEFI allocators

[tsokano]

Since this commit, Thread.spawn() (which worked until 8ad0732) crashes on Android (aarch64-linux-android), using std.heap.c_allocator.
The same applies to ddff1fa.

[alexrp]

Since this commit, Thread.spawn() (which worked until 8ad0732) crashes on Android (aarch64-linux-android), using std.heap.c_allocator. The same applies to ddff1fa.

Please file a bug with a repro so this doesn't get lost.