Two-phase disk cache (archives + installed) with LRU GC, sized budgets, and crash-safe SQLite index

[zackees]

Summary

Replace the current free-form cache layout in crates/fbuild-packages/src/cache.rs with a two-phase disk cache that separates what we downloaded from what we installed, plus a background LRU GC that runs against configurable size budgets. The whole system needs a single source of truth for paths and a crash-safe index that survives daemon restarts and concurrent access.

Today we only have Cache (crates/fbuild-packages/src/cache.rs) which exposes path resolution but no awareness of disk usage, no LRU tracking, and no notion of "archive vs installed". Toolchains, platforms, packages, and libraries grow without bound.

Goals

• Bound disk usage with sane, auto-scaled defaults.
• Keep cheap-to-rehydrate state (extracted dirs) evictable while preserving the expensive-to-fetch state (archive blobs).
• Make GC safe to run concurrently with builds (don't yank a toolchain mid-compile).
• Survive a kill -9 of the singleton daemon — index must never be left in a torn state.
• Centralize path/budget logic so platform orchestrators never hand-roll cache paths.

Proposed Layout

Two physically separate trees under ~/.fbuild/{dev|prod}/cache/:

cache/
 archives/ # PHASE 1 — original downloads, never re-downloadable for free
 {kind}/{stem}/{hash}/{version}/{filename}.{tar.gz|zip|xz|...}
 {kind}/{stem}/{hash}/{version}/.sha256
 installed/ # PHASE 2 — extracted/installed, regeneratable from archives
 {kind}/{stem}/{hash}/{version}/<contents>
 {kind}/{stem}/{hash}/{version}/.install_complete
 index.sqlite # crash-safe LRU + reference index
 index.sqlite-wal
 gc.lock # advisory lock for GC critical sections

{kind} is one of packages, toolchains, platforms, libraries, frameworks. Stem/hash/version semantics stay the same as today (url_stem + first 16 chars of SHA256 + version), so existing call sites mostly just change which subtree they read from.

The split makes the eviction policy obvious: GC always purges from installed/ first, archives only when they exceed their own budget.

Index Design (SQLite)

Single-file SQLite DB at ~/.fbuild/{mode}/cache/index.sqlite, opened in WAL mode with synchronous=NORMAL. WAL gives us multi-reader/single-writer safely across processes and survives crashes (the WAL is replayed on next open).

Schema sketch:

CREATE TABLE entries (
 id INTEGER PRIMARY KEY,
 kind TEXT NOT NULL, -- packages|toolchains|platforms|libraries|frameworks
 url TEXT NOT NULL,
 stem TEXT NOT NULL,
 hash TEXT NOT NULL, -- 16-char sha256 prefix
 version TEXT NOT NULL,
 archive_path TEXT, -- relative to cache root, NULL until downloaded
 archive_bytes INTEGER,
 archive_sha256 TEXT,
 installed_path TEXT, -- relative, NULL until extracted
 installed_bytes INTEGER,
 installed_at INTEGER, -- unix epoch
 archived_at INTEGER,
 last_used_at INTEGER NOT NULL, -- driven by touch() on every cache hit
 use_count INTEGER NOT NULL DEFAULT 0,
 pinned INTEGER NOT NULL DEFAULT 0, -- in-use protection (see leases below)
 UNIQUE(kind, hash, version)
);
CREATE INDEX idx_lru_installed ON entries(last_used_at) WHERE installed_path IS NOT NULL;
CREATE INDEX idx_lru_archive ON entries(last_used_at) WHERE archive_path IS NOT NULL;

CREATE TABLE leases (
 entry_id INTEGER NOT NULL REFERENCES entries(id) ON DELETE CASCADE,
 holder_pid INTEGER NOT NULL,
 acquired_at INTEGER NOT NULL,
 PRIMARY KEY(entry_id, holder_pid)
);

Why SQLite over a hand-rolled index:

• Atomic transactions for "add entry + record bytes + bump LRU" — no torn writes if the daemon dies mid-update.
• WAL = multi-process safe for free, including the CLI hitting the cache without going through the daemon.
• pinned/leases model lets us hold a reference for the duration of a build without copying the directory.
• Replaces the implicit "directory exists therefore it is cached" check, which is wrong the moment an extraction fails halfway.

Crash-safety contract:

1. Download: write to archives/.../<file>.partial, fsync, rename to final, then INSERT OR REPLACE row. If we die before the row insert, GC sweeps the orphan via reconciliation pass (see below).
2. Install: extract into installed/.../<dir>.partial, write .install_complete sentinel, rename, then update row. Same orphan sweep handles partial extracts.
3. Reconciliation on daemon start: walk archives/ and installed/, cross-check against entries. Orphan files → delete. Orphan rows (path missing on disk) → null out the column. This is what makes "singleton daemon killed and restarted" safe: the index is the truth, but it is reconciled against the filesystem on every startup, so neither side can drift permanently.

GC Policy

Background task in fbuild-daemon, scheduled every N minutes (configurable, default 5min) and also kicked after every successful build. Single GC instance enforced via gc.lock (same advisory-lock pattern the daemon already uses for its in-memory managers — no new file-lock primitive).

Budgets (auto-scaled at startup, overridable via env):

ARCHIVE_BUDGET = min(15 GiB, 5% of total disk)
INSTALLED_BUDGET = min(15 GiB, 5% of total disk)
HIGH_WATERMARK = min(30 GiB, 10% of total disk) # combined trigger
LOW_WATERMARK = 80% of HIGH_WATERMARK # GC stops here

Env overrides: FBUILD_CACHE_ARCHIVE_BUDGET, FBUILD_CACHE_INSTALLED_BUDGET, FBUILD_CACHE_HIGH_WATERMARK. All accept human sizes (15G, 500M).

Eviction order (cheap → expensive):

1. If combined size > HIGH_WATERMARK or installed/ > INSTALLED_BUDGET: evict installed/ directories LRU-first, skipping any entry with pinned > 0 or an active lease, until installed-bytes ≤ LOW_WATERMARK. Only the directory is removed; the row stays so the next user can re-extract from the archive.
2. If archives/ > ARCHIVE_BUDGET: evict archive files LRU-first, again skipping leased entries. Once an archive is gone and there is no installed dir, the row is deleted.
3. Never delete an entry that is currently leased by a live PID. Stale leases (PID dead) are reaped at the start of each GC cycle.

This gives the property requested: "unused installed folders go first, archives only when they too get fat."

Centralized Path + Budget Module

New module crates/fbuild-packages/src/disk_cache/ (replaces the current cache.rs over time):

disk_cache/
 mod.rs # public DiskCache facade
 paths.rs # SOLE source of cache paths (archives_dir, installed_dir, entry_path, etc.)
 index.rs # SQLite open/migrate/query/touch/reconcile
 budget.rs # size accounting + watermark math + auto-scaling from disk space
 gc.rs # eviction loop, lease reaping, lock handling
 lease.rs # RAII Lease guard — bumps `pinned`, releases on drop

Public surface (everything else in the workspace goes through this):

pub struct DiskCache { /* ... */ }

impl DiskCache {
 pub fn open() -> Result<Self>; // opens/migrates index
 pub fn lookup(&self, kind: Kind, url: &str, version: &str) -> Option<CacheEntry>;
 pub fn record_archive(&self, ...) -> Result<CacheEntry>; // after download
 pub fn record_install(&self, ...) -> Result<CacheEntry>; // after extract
 pub fn lease(&self, entry: &CacheEntry) -> Lease; // holds pin until drop
 pub fn touch(&self, entry: &CacheEntry); // bump LRU
 pub fn run_gc(&self) -> GcReport; // manual trigger
 pub fn stats(&self) -> CacheStats; // for `fbuild status`
}

Callers (toolchain installer, library manager, platform downloader) must:

1. lookup first.
2. If miss → download to staging path from paths::archive_staging(...), then record_archive.
3. Extract to paths::install_staging(...), then record_install.
4. Hold Lease for the duration of any build that uses the entry.

This kills the existing pattern of every orchestrator constructing its own paths via Cache::get_toolchain_path etc.

Migration

fbuild-packages::cache::Cache becomes a thin shim over DiskCache for one release, logging a deprecation warning when constructed. Existing on-disk layout (packages/, toolchains/, platforms/, libraries/ directly under cache/) is migrated on first daemon start: move into installed/<kind>/..., leave archives untouched (we do not have them yet), insert rows with last_used_at = now. Migration is idempotent and gated on a cache_schema_version row in the index.

Tests (TDD — these go first)

Each test uses tempfile::TempDir, no mocks (per CLAUDE.md test policy):

• test_index_open_creates_schema — fresh dir → schema migrated, version row present.
• test_record_archive_then_install_roundtrip — entry visible to lookup.
• test_reconcile_orphan_file_deleted — drop a file in archives/ with no row → next open removes it.
• test_reconcile_orphan_row_nulled — row points at missing path → column nulled, not deleted (entry may still have other phase).
• test_lease_blocks_eviction — held lease prevents GC from removing the dir/file.
• test_dead_pid_lease_reaped — lease for a non-existent PID is dropped on next GC.
• test_gc_evicts_installed_first — installed > budget, archive within budget → only installed evicted.
• test_gc_evicts_archives_when_over_budget — both over budget → installed first, then archives.
• test_gc_low_watermark_stops_eviction — GC stops at LOW_WATERMARK, does not over-evict.
• test_budget_autoscales_to_disk — small disk → budgets clamp to 5%/10% rather than the absolute caps.
• test_concurrent_lookup_and_gc — two threads, one looking up + leasing, one running GC — leased entry survives.
• test_crash_during_install_recovers — .install_complete is missing → reconcile removes the partial dir.
• test_kill_dash_9_simulation — open index, write a partial transaction via a second connection, drop without commit, reopen → WAL replay leaves a consistent state.
• test_migration_from_legacy_layout — pre-populate old cache/toolchains/... layout → first open moves files into installed/toolchains/... and indexes them.

Stress test (gated behind --full):

• 10k entries, randomized lookups + inserts + GC over 60s, assert no torn rows and combined size stays at or below HIGH_WATERMARK.

Open Questions

1. rusqlite vs sqlx: rusqlite is sync, smaller, and matches the daemon's existing blocking-thread model for fs work. sqlx is async-native but pulls in a much heavier surface. Leaning rusqlite + tokio::task::spawn_blocking for the few async call sites.
2. Per-entry checksums on read: do we re-verify the archive sha256 on every cache hit, or only on download + GC pass? Probably the latter (cost outweighs value on the hot path).
3. Cross-mode sharing: dev and prod each get their own index today. Should installed toolchains be shareable across modes via symlink to save 5–10 GiB on dev machines? Out of scope for v1, worth a follow-up issue.
4. Eviction granularity: evict whole (kind, hash, version) or per-file? Whole-entry is simpler and matches how callers consume them. Going with whole-entry.

Acceptance Criteria

• DiskCache module exists and is the only producer of cache paths in the workspace.
• All existing call sites in fbuild-packages, fbuild-build, and platform orchestrators route through DiskCache::lookup / record_* / lease.
• SQLite index opens in WAL mode, migrates cleanly, reconciles on every daemon start.
• Background GC honors budgets, watermarks, leases, and the eviction order above.
• kill -9 of the daemon mid-write leaves the index recoverable on next start (covered by test).
• Auto-scaled budgets default to min(15G, 5% disk) per phase and min(30G, 10% disk) combined.
• Legacy on-disk layout is migrated automatically and idempotently.
• fbuild status reports current cache size, budget, and number of pinned entries.
• fbuild purge --gc triggers a synchronous GC pass and prints a GcReport.

Files Likely Touched

• crates/fbuild-packages/src/cache.rs → split into disk_cache/ module
• crates/fbuild-packages/Cargo.toml (add rusqlite with bundled feature)
• crates/fbuild-paths/src/lib.rs (cache subpath helpers — archives_root, installed_root, index_path)
• crates/fbuild-daemon/src/... (spawn GC loop on startup, expose /cache/stats, /cache/gc)
• crates/fbuild-cli/src/... (fbuild status, fbuild purge --gc)
• crates/fbuild-build/src/... orchestrators (replace direct Cache::get_* calls with DiskCache + Lease)