Skip to content

Commit 23526ac

Browse files
tools: add a guarded Zig-cache inspection/cleanup tool + docs (#53)
Repeated focused test builds relink the root test artifact under distinct cache keys, and the repo-local `.zig-cache` has reached ~92 GB, tripping LLVM `No space left on device`. Everything in `.zig-cache`/`zig-out` is reproducible, so add a safe way to inspect and reclaim it. `tools/zig-cache-tool.sh`: - `report` (default): total + per-subdir sizes, largest o/ artifacts, and the reclaimable total. - `prune [--dry-run] [--all]`: remove the reproducible bulk (o/, tmp/, zig-out; +h/ z/ with --all). It accepts NO path arguments — every target is a fixed repository-relative directory — and before each removal it resolves the target through symlinks and refuses anything not inside `<repo>/.zig-cache` or `<repo>/zig-out` (path-traversal like `.zig-cache/../src` is rejected). It finds the repo from its own location via `git rev-parse`, so it never acts elsewhere. Adds `docs/dev-cache.md` explaining inspection and safe cleanup, linked from the baseline-JIT dev doc. Covers issue #53 acceptance criteria for the documented, guarded cache-cleanup path; the focused per-module test artifacts (criteria 1-3) intersect the active build.zig/JIT restructuring and are left to that work.
1 parent 971f994 commit 23526ac

3 files changed

Lines changed: 213 additions & 0 deletions

File tree

docs/baseline-jit.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,3 +161,6 @@ Performance evidence uses the symmetric JSC protocol in
161161
the 1,232-sample publication matrix is rerun only after a meaningful batch of
162162
optimizations. A speedup is not accepted if checksums, supported rows, or
163163
execution accounting differ.
164+
165+
Repeated focused test builds grow the reproducible `.zig-cache` quickly; see
166+
[`dev-cache.md`](dev-cache.md) for inspecting and safely reclaiming it.

docs/dev-cache.md

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Inspecting and cleaning the Zig build cache
2+
3+
Zig writes all build output into two repository-local directories:
4+
5+
| Path | Contents |
6+
| ------------- | --------------------------------------------------------------- |
7+
| `.zig-cache/` | compiled objects (`o/`), scratch (`tmp/`), hashes (`h/`, `z/`) |
8+
| `zig-out/` | installed artifacts (`zig build` `install` step outputs) |
9+
10+
**Everything in both directories is reproducible** — deleting them only forces
11+
the next build to recompile. Neither ever contains source or user-owned files.
12+
13+
## Why it grows without bound
14+
15+
Each distinct build configuration produces a distinct cache key and a fresh copy
16+
of the artifact. Repeated *focused* test runs are the usual culprit: a filtered
17+
build such as
18+
19+
```sh
20+
zig build test -Dtest-filter=vm # then ...=jit, ...=compiler, ...
21+
```
22+
23+
relinks the whole root test binary under a new key for every filter string, so a
24+
day of baseline-JIT development can leave dozens of multi-hundred-MB artifacts
25+
behind. In practice `.zig-cache` has reached **~92 GB**, at which point LLVM
26+
fails with `No space left on device`. Clearing the reproducible cache recovers
27+
all of it.
28+
29+
## Inspect
30+
31+
```sh
32+
tools/zig-cache-tool.sh report # (default) sizes, per-subdir breakdown,
33+
# largest o/ artifacts, reclaimable total
34+
```
35+
36+
## Clean
37+
38+
```sh
39+
tools/zig-cache-tool.sh prune --dry-run # preview exactly what would be removed
40+
tools/zig-cache-tool.sh prune # remove .zig-cache/o, .zig-cache/tmp, zig-out
41+
tools/zig-cache-tool.sh prune --all # additionally drop h/ and z/ (fully cold rebuild)
42+
```
43+
44+
`prune` removes only the reproducible bulk (compiled objects, scratch, and the
45+
install tree) and keeps the small hash/manifest dirs so the next build can still
46+
reuse unchanged inputs; `--all` forces a completely cold rebuild.
47+
48+
## Safety
49+
50+
The tool is deliberately conservative and is the recommended way to clean the
51+
cache:
52+
53+
- It accepts **no path arguments** — every target is a fixed repository-relative
54+
directory (`.zig-cache/o`, `.zig-cache/tmp`, `.zig-cache/h`, `.zig-cache/z`,
55+
`zig-out`), so there is no way to point it at anything else.
56+
- Before each removal it resolves the target through symlinks and refuses to
57+
proceed unless the result is inside `<repo>/.zig-cache` or `<repo>/zig-out`.
58+
Path-traversal attempts (e.g. a symlinked cache pointing outside the repo)
59+
are rejected rather than followed.
60+
- It locates the repository from its own location via `git rev-parse`, so it
61+
never acts on an unrelated directory even if run from elsewhere.
62+
63+
The manual equivalent is `rm -rf .zig-cache zig-out` from the repo root, but the
64+
tool adds the reporting and the outside-the-repo guard, and never leaves the
65+
hash dirs in an inconsistent partial state.

tools/zig-cache-tool.sh

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
#!/usr/bin/env bash
2+
#
3+
# zig-cache-tool.sh — inspect and safely prune the repository-local Zig caches.
4+
#
5+
# During baseline-JIT development, repeated focused `zig build ... -Dtest-filter`
6+
# runs relink the monolithic test artifact under distinct cache keys and the
7+
# repo-local `.zig-cache` has grown to ~92 GB, tripping LLVM `No space left on
8+
# device`. This tool reports what those reproducible caches hold and prunes them
9+
# without ever being able to touch source or user-owned files (issue #53).
10+
#
11+
# Usage:
12+
# tools/zig-cache-tool.sh report # (default) sizes + largest entries
13+
# tools/zig-cache-tool.sh prune [--dry-run] # remove the reproducible cache (o/ tmp/) + zig-out
14+
# tools/zig-cache-tool.sh prune --all [...] # also remove h/ and z/ (hashes/manifests)
15+
#
16+
# Only ever operates on <repo>/.zig-cache and <repo>/zig-out — both are fully
17+
# reproducible from a build. It refuses any path that, after symlink resolution,
18+
# is not inside those two directories.
19+
20+
set -euo pipefail
21+
22+
# --- Locate the repository, whatever CWD the tool is invoked from. ------------
23+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
24+
REPO_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || true)"
25+
if [ -z "$REPO_ROOT" ]; then
26+
echo "error: not inside a git repository (run from the zig-js checkout)" >&2
27+
exit 1
28+
fi
29+
CACHE_DIR="$REPO_ROOT/.zig-cache"
30+
OUT_DIR="$REPO_ROOT/zig-out"
31+
32+
# --- Safety guard: refuse anything outside the two reproducible output dirs. --
33+
# Resolves the target's parent through symlinks (the target itself may not
34+
# exist), then requires the absolute result to be exactly, or a child of,
35+
# CACHE_DIR or OUT_DIR. This is the hard boundary for issue #53's rule that no
36+
# cleanup can target paths outside the repository cache/build outputs.
37+
assert_within_outputs() {
38+
local target="$1" parent base resolved
39+
parent="$(dirname "$target")"
40+
base="$(basename "$target")"
41+
if ! resolved="$(cd "$parent" 2>/dev/null && printf '%s/%s' "$(pwd -P)" "$base")"; then
42+
echo "refusing: cannot resolve '$target'" >&2
43+
return 1
44+
fi
45+
case "$resolved" in
46+
"$CACHE_DIR" | "$CACHE_DIR"/* | "$OUT_DIR" | "$OUT_DIR"/*) return 0 ;;
47+
*)
48+
echo "refusing to touch a path outside the repo cache/build outputs:" >&2
49+
echo " $resolved" >&2
50+
return 1
51+
;;
52+
esac
53+
}
54+
55+
human_size() {
56+
# Total apparent size of the arguments that exist, as a human string.
57+
local existing=()
58+
for p in "$@"; do [ -e "$p" ] && existing+=("$p"); done
59+
if [ ${#existing[@]} -eq 0 ]; then
60+
echo "0B"
61+
else
62+
du -sh -c "${existing[@]}" 2>/dev/null | tail -1 | cut -f1
63+
fi
64+
}
65+
66+
cmd_report() {
67+
echo "repository: $REPO_ROOT"
68+
echo
69+
if [ -d "$CACHE_DIR" ]; then
70+
echo ".zig-cache total: $(human_size "$CACHE_DIR")"
71+
for sub in o tmp h z; do
72+
[ -d "$CACHE_DIR/$sub" ] && printf ' %-4s %s\n' "$sub/" "$(human_size "$CACHE_DIR/$sub")"
73+
done
74+
if [ -d "$CACHE_DIR/o" ]; then
75+
echo
76+
echo " largest o/ artifacts (compiled test/exe outputs — the reproducible bulk):"
77+
du -sh "$CACHE_DIR"/o/* 2>/dev/null | sort -rh | head -8 | sed 's/^/ /'
78+
fi
79+
else
80+
echo ".zig-cache: (none)"
81+
fi
82+
echo
83+
if [ -d "$OUT_DIR" ]; then
84+
echo "zig-out total: $(human_size "$OUT_DIR")"
85+
else
86+
echo "zig-out: (none)"
87+
fi
88+
echo
89+
echo "reclaimable now: $(human_size "$CACHE_DIR/o" "$CACHE_DIR/tmp" "$OUT_DIR")"
90+
echo "run 'tools/zig-cache-tool.sh prune' to reclaim it."
91+
}
92+
93+
cmd_prune() {
94+
local dry_run=0 all=0
95+
for arg in "$@"; do
96+
case "$arg" in
97+
--dry-run) dry_run=1 ;;
98+
--all) all=1 ;;
99+
*) echo "unknown prune option: $arg" >&2; exit 2 ;;
100+
esac
101+
done
102+
103+
# The reproducible bulk: compiled outputs (o/), scratch (tmp/), and zig-out.
104+
# --all additionally drops the small hash (h/) and manifest (z/) dirs, forcing
105+
# a fully cold rebuild.
106+
local targets=("$CACHE_DIR/o" "$CACHE_DIR/tmp" "$OUT_DIR")
107+
if [ "$all" -eq 1 ]; then
108+
targets+=("$CACHE_DIR/h" "$CACHE_DIR/z")
109+
fi
110+
111+
local reclaim
112+
reclaim="$(human_size "${targets[@]}")"
113+
echo "would reclaim: $reclaim"
114+
115+
for t in "${targets[@]}"; do
116+
assert_within_outputs "$t" || exit 1
117+
if [ ! -e "$t" ]; then
118+
continue
119+
fi
120+
if [ "$dry_run" -eq 1 ]; then
121+
echo " [dry-run] rm -rf $t"
122+
else
123+
echo " removing $t"
124+
rm -rf -- "$t"
125+
fi
126+
done
127+
128+
if [ "$dry_run" -eq 1 ]; then
129+
echo "dry run: nothing was deleted."
130+
else
131+
echo "done. reclaimed ~$reclaim (rebuilds are fully reproducible)."
132+
fi
133+
}
134+
135+
case "${1:-report}" in
136+
report) cmd_report ;;
137+
prune) shift; cmd_prune "$@" ;;
138+
-h | --help | help)
139+
sed -n '2,25p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
140+
;;
141+
*)
142+
echo "unknown command: $1 (expected: report | prune | help)" >&2
143+
exit 2
144+
;;
145+
esac

0 commit comments

Comments
 (0)