audit: blocking operations with no timeout in fbuild-build (sub-issue of #802)

[zackees]

Intro

Audited crates/fbuild-build/ for blocking operations that have no
timeout / deadline / cancellation. Scope: build orchestration, per-platform
orchestrators (AVR, ESP32, ESP8266, RP2040, STM32, Teensy, SAM, NRF52, CH32V,
Renesas, Silabs, NXP-LPC, Apollo3, generic-arm), the shared compile pipeline,
linker invocations, the embedded zccache compile dispatch, the extra_scripts
Python harness, and the symbol/bloat analyzers (nm, objdump, c++filt,
size, objcopy, ar).

The dominant failure mode in this crate is subprocess invocations of toolchain
binaries via fbuild_core::subprocess::run_command(..., timeout: None).
That helper does support a timeout: Option<Duration> (with Timeout error
mapping and process.kill() on expiry — see crates/fbuild-core/src/subprocess.rs
lines 81-208), so the fix at every site below is mechanical: pass
Some(Duration::from_secs(N)) with an N chosen to fit the operation
(5-30s for utility tools like nm/size/objcopy/-dumpversion, longer for
actual link steps that legitimately run for minutes on large ESP32 firmware).

A handful of utility sites already do this correctly (e.g. esp32_linker.rs:381
elf2image with 30s, boot_artifacts.rs:94/145, shrink/probe.rs:352,
esp8266_linker.rs:262) — they should be the template for the missing ones.

The second class of finding is the embedded zccache dispatch in
zccache_embedded.rs / compile_backend.rs. Every per-translation-unit compile
flows through FbuildZccacheService::compile_blocking, which calls
tokio::runtime::Handle::block_on(svc.compile(req).await) with no
tokio::time::timeout(...) wrapper. If the upstream ZccacheService::compile
ever hangs (network-mode cache backends, deadlocked fingerprint scan, etc.)
the build worker thread parks forever. This is the single highest-impact
finding because it is on the hot compile path inside the daemon.

The third class is the extra_scripts Python harness invocation in
script_runtime.rs. User-supplied extra_scripts = [...] in platformio.ini
are evaluated by spawning Python with no timeout — a hostile or buggy script
can wedge the daemon's build pipeline indefinitely.

Findings

Severity	File:line	Operation	Why unbounded	Suggested fix
CRITICAL	crates/fbuild-build/src/zccache_embedded.rs:215-217	runtime.block_on(async move { inner.compile(req).await }) inside FbuildZccacheService::compile_blocking	Every per-TU compile (called from compiler.rs:665 via compile_source) blocks on this with no tokio::time::timeout. Hangs in the zccache backend stall a daemon HTTP worker until the user kills the process.	Wrap with tokio::time::timeout(Duration::from_secs(300), inner.compile(req)), return EmbeddedServiceError::Timeout, surface as FbuildError::Timeout at the compiler.rs:665 call site. 5 min is a reasonable upper bound for any single compile.
CRITICAL	crates/fbuild-build/src/zccache_embedded.rs:150-152	ZccacheService::start(cfg).await in start_in	Called once from the daemon's #[tokio::main] via CompileBackend::start (compile_backend.rs:48) before any HTTP listener comes up. If start hangs (e.g. corrupt cache_root, identity hash livelock) the daemon never finishes startup and never logs a useful error. Affects 100% of users.	Wrap with tokio::time::timeout(Duration::from_secs(30), ZccacheService::start(cfg)).
CRITICAL	crates/fbuild-build/src/zccache_embedded.rs:227-230	FbuildZccacheService::flush().await	Called from shutdown path and (via FbuildZccacheService) potentially from daemon shutdown hooks. Unbounded flush blocks daemon shutdown.	Wrap with tokio::time::timeout(Duration::from_secs(15), self.inner.flush()).
CRITICAL	crates/fbuild-build/src/zccache_embedded.rs:250-256	ZccacheService::shutdown(mode).await	Same risk as flush, plus inability to terminate the daemon cleanly.	Wrap with tokio::time::timeout(Duration::from_secs(30), svc.shutdown(mode)).
HIGH	crates/fbuild-build/src/script_runtime.rs:111	run_command(&argv, Some(project_dir), None, None) running user extra_scripts Python harness	User-supplied Python code; a buggy/malicious extra_scripts entry hangs the build. Hot path for any project using extra_scripts in platformio.ini.	Pass Some(Duration::from_secs(60)) — extra_scripts is config-time evaluation, never legitimately long.
HIGH	crates/fbuild-build/src/script_runtime.rs:297	run_command(&argv, None, None, None) running python --version for python discovery	Trivial command but on the startup path. A hung Python interpreter wedges build init.	Some(Duration::from_secs(5)).
HIGH	crates/fbuild-build/src/compiler.rs:438	run_command(&[gcc, "-dumpversion"], None, None, None) in compiler_version	Trivial command, but cached per compiler path and called on every fresh build. A wedged toolchain binary (corrupt EXE, missing DLL hang on Windows) blocks the whole pipeline.	Some(Duration::from_secs(5)).
HIGH	crates/fbuild-build/src/pipeline/compile.rs:167-172	run_command(&[gcc, "-dumpversion"], ...) in log_toolchain_version	Same risk as compiler_version — gcc -dumpversion should never block.	Some(Duration::from_secs(5)).
HIGH	crates/fbuild-build/src/linker.rs:331	run_command(&args_ref, None, None, None) — ar rcs <archive> <objects...> in LinkerBase::archive	Used by every platform (avr, esp32, esp8266, nrf52, sam, silabs, teensy, etc.) to build the framework core archive. ar can occasionally hang on very large arg lists on Windows.	Some(Duration::from_secs(120)).
HIGH	crates/fbuild-build/src/linker.rs:359	run_command(...) — <prefix>-size firmware.elf in LinkerBase::report_size	Final-stage size report on the build.	Some(Duration::from_secs(15)).
HIGH	crates/fbuild-build/src/linker.rs:409	run_command(...) — <prefix>-nm --print-size --size-sort ... in LinkerBase::analyze_symbols	Symbol map analysis at end of build.	Some(Duration::from_secs(30)).
HIGH	crates/fbuild-build/src/linker.rs:455	run_command(...) — <prefix>-objcopy -O <fmt> <elf> <hex> in LinkerBase::objcopy_firmware	ELF to HEX/BIN conversion used by AVR and Teensy.	Some(Duration::from_secs(60)).
HIGH	crates/fbuild-build/src/avr/avr_linker.rs:187	run_command(...) — main avr-gcc link step	Hot path: the primary link command for every AVR build.	Some(Duration::from_secs(180)).
HIGH	crates/fbuild-build/src/ch32v/ch32v_linker.rs:113	run_command(...) — main riscv-none-elf-gcc link step	Hot path for CH32V builds.	Some(Duration::from_secs(180)).
HIGH	crates/fbuild-build/src/esp32/esp32_linker.rs:325	run_command(...) — main ESP32 link (Windows response-file path)	Hot path for ESP32 link on Windows. Already has a 30s sibling at line 381 for elf2image — the link itself should also have one.	Some(Duration::from_secs(300)) — ESP32 links can be long.
HIGH	crates/fbuild-build/src/esp32/esp32_linker.rs:328	run_command(...) — main ESP32 link (non-Windows path)	Same as above.	Some(Duration::from_secs(300)).
HIGH	crates/fbuild-build/src/esp32/orchestrator/embed.rs:72	run_command(...) — objcopy for board_build.embed_files binary embedding	Per-file in user's embed_files list.	Some(Duration::from_secs(30)).
HIGH	crates/fbuild-build/src/esp32/orchestrator/embed.rs:132	run_command(...) — objcopy for board_build.embed_txtfiles	Same as above for text embedding.	Some(Duration::from_secs(30)).
HIGH	crates/fbuild-build/src/esp8266/esp8266_linker.rs:121	run_command(...) — gcc -E -P linker-script preprocessing	Build-time preprocessor invocation.	Some(Duration::from_secs(30)).
HIGH	crates/fbuild-build/src/esp8266/esp8266_linker.rs:228	run_command(...) — main xtensa-lx106-elf-gcc link step	Hot path for ESP8266 builds.	Some(Duration::from_secs(180)).
HIGH	crates/fbuild-build/src/nrf52/nrf52_linker.rs:128	run_command(...) — main link step	Hot path for nRF52 builds.	Some(Duration::from_secs(180)).
HIGH	crates/fbuild-build/src/sam/sam_linker.rs:149	run_command(...) — main link step	Hot path for SAM/Due builds.	Some(Duration::from_secs(180)).
HIGH	crates/fbuild-build/src/renesas/renesas_linker.rs:133	run_command(...) — main link step	Hot path for Renesas (UNO R4) builds.	Some(Duration::from_secs(180)).
HIGH	crates/fbuild-build/src/generic_arm/arm_linker.rs:173	run_command(...) — main link (Windows response-file path)	Hot path for generic ARM builds.	Some(Duration::from_secs(180)).
HIGH	crates/fbuild-build/src/generic_arm/arm_linker.rs:176	run_command(...) — main link (non-Windows path)	Same as above.	Some(Duration::from_secs(180)).
HIGH	crates/fbuild-build/src/teensy/teensy_linker.rs:166	run_command(...) — main link (Windows response-file path)	Hot path for Teensy. teensy41 has ~500 .o files.	Some(Duration::from_secs(180)).
HIGH	crates/fbuild-build/src/teensy/teensy_linker.rs:169	run_command(...) — main link (non-Windows path)	Same as above.	Some(Duration::from_secs(180)).
HIGH	crates/fbuild-build/src/silabs/silabs_linker.rs:136	run_command(...) — main link step	Hot path for SiLabs builds.	Some(Duration::from_secs(180)).
HIGH	crates/fbuild-build/src/stm32/orchestrator/arduino_mbed.rs:272	run_command(...) — g++ -E -P linker-script preprocessing for STM32 Arduino-mbed variant	Build-time preprocessor invocation.	Some(Duration::from_secs(30)).
MEDIUM	crates/fbuild-build/src/symbol_analyzer/mod.rs:178	run_command_with_stdin(&[cppfilt], stdin, ..., None) in demangle_batch	End-of-build bloat analysis. c++filt is generally well-behaved but the stdin payload can be hundreds of KB; a wedged child would freeze the symbol report.	Some(Duration::from_secs(60)).
MEDIUM	crates/fbuild-build/src/symbol_analyzer/mod.rs:228	run_command(&[nm, "--print-size", ...], None, None, None) in analyze_elf	End-of-build symbol analysis.	Some(Duration::from_secs(60)).
MEDIUM	crates/fbuild-build/src/symbol_analyzer/mod.rs:370	run_command(&[objdump, "-d", "--no-show-raw-insn", elf], ..., None) for callgraph extraction	End-of-build, off the critical path but still daemon-thread. objdump -d on a large ESP32 ELF can produce 100+ MB of output.	Some(Duration::from_secs(120)).

What was searched

Patterns run via Grep over crates/fbuild-build/src/:

• Command::new — no production matches (only test files in crates/fbuild-build/tests/)
• \.output\(\) — only tests
• \.wait\(\)|\.wait_with_output\(\)|wait_timeout — only Barrier::wait() in tests
• reqwest|TcpStream|TcpListener — no matches in src/ (only Cargo.toml dep)
• tokio::time::timeout|tokio::time::sleep|tokio::process — no matches in this crate
• \.await — only in compile_backend.rs and zccache_embedded.rs (audited individually above)
• \.lock\(\) — all are std::sync::Mutex::lock(), scoped to local data structures (work_iter, first_error, results_slot, build_log, COMPILER_IDENTITY_CACHE, fingerprint cache entries), not external state. No deadlock risk identified — locks are held only across small map/insert/push ops, never across await points or subprocess waits.
• \.read\(\)\.unwrap|\.write\(\)\.unwrap — no matches (no RwLock use)
• \.recv\(\)|recv_timeout|mpsc:: — only mpsc::Sender in build_output.rs and lib.rs (no receivers in this crate; receiver lives in fbuild-daemon's websocket pump and is out of scope)
• thread::sleep|Instant::now|Duration::from — Instant::now is everywhere for perf timing (not for polling). thread::sleep only in test code.
• block_on — single production occurrence in zccache_embedded.rs:216 (covered above).
• run_command\( / run_command_with_stdin / run_command_passthrough — every call site catalogued. Sites with Some(Duration::...) already (good template):
  • esp32_linker.rs:381 — 30s for esptool elf2image
  • esp8266_linker.rs:262 — 30s for esptool elf2image
  • esp32/orchestrator/boot_artifacts.rs:94 — 30s for bootloader elf2image
  • esp32/orchestrator/boot_artifacts.rs:145 — 10s for gen_esp32part.py
  • shrink/probe.rs:352 — 5s for probe compile

Files spot-checked for full context:

• crates/fbuild-core/src/subprocess.rs — confirmed run_command honors timeout via process.wait(timeout) and kills the child on ProcessError::Timeout.
• crates/fbuild-build/src/compiler.rs — confirmed compile_source is the single funnel from every platform compiler module into compile_blocking.
• crates/fbuild-build/src/parallel.rs — confirmed the loop {} in the worker terminates when the work_iter runs out; not a busy-poll.
• crates/fbuild-build/src/build_fingerprint/fast_path.rs — Mutex use is bounded to local map operations.

Out-of-scope notes

• crates/fbuild-build/tests/*.rs and *_tests.rs modules — test code with Command::new, Barrier::wait(), and thread::sleep for synchronization. These can hang CI if they break but the fix is per-test (most already use bounded sleeps under 50ms).
• Drop impls — no Drop impls in this crate perform blocking IO. (The embedded ZccacheService Drop is upstream in zccache and out of scope per audit: blocking operations with no timeout (meta) #802's audit guidance.)
• running_process crate internals — the subprocess primitive itself has timeout support; bugs in its implementation are out of scope for this audit.
• fbuild-daemon HTTP handlers — out of scope (separate auditor area).
• compile_database/ — pure JSON serialization; no blocking IO besides std::fs which is filesystem-bounded.
• eh_frame_policy*.rs, flag_overlay.rs, arduino_props.rs, package_override.rs, framework_libs.rs, framework_core_cache.rs, resolution.rs, lib.rs — pure config / data transformation, no subprocess or locking concerns.
• source_scanner/ — uses io::stderr().lock() for line-by-line output flushing; not unbounded blocking.

Sub-issue of #802.

[zackees]

Closed by PR #843 (workspace-wide timeout sweep).