Speculative Actions: cache priming via adapted subactions by sstriker · Pull Request #2109 · apache/buildstream

sstriker · 2026-03-23T21:56:44Z

Closes#2083

Summary

Speculative actions speed up rebuilds by pre-populating the action cache with adapted versions of previously recorded build actions. When a dependency changes, compile and link commands from the previous build are adapted with updated input digests and executed ahead of the actual build, so recc gets action cache hits instead of executing from scratch.

Recording: subactions captured via recc through remote-apis-socket during sandbox execution
Generation: after each build, overlays describe how each subaction's inputs relate to source elements and dependency artifacts
Storage: speculative actions stored on artifact proto under the weak cache key (stable across dependency version changes)
Priming: before each rebuild, stored actions are instantiated with current dependency digests and submitted fire-and-forget
Tiered modes: none/prime-only/source-artifact/intra-element/full — lets users control cost vs benefit

See doc/source/arch_speculative_actions.rst for the full architecture documentation including example scenarios.

Dependencies

Depends on: https://gitlab.com/BuildGrid/buildbox/buildbox/-/merge_requests/1080

Test plan

46 unit tests (generator, instantiator, pipeline integration, weak key, modes)
5 integration tests with real sandbox + recc + buildbox-casd
Full test suite: 1753 passed, 0 failed
End-to-end with freedesktop-sdk sstriker/recc

🤖 Generated with Claude Code

Proto definitions: - speculative_actions.proto: SpeculativeActions, SpeculativeAction, Overlay messages for storing build action overlays - ActionResult.subactions (field 99): repeated Digest for nested execution action digests recorded by recc/trexe - Artifact.speculative_actions: Digest field for SA proto storage Generator (generator.py): - Analyzes completed builds to extract subaction digests from Actions - Builds digest cache mapping file hashes to source elements (SOURCE priority > ARTIFACT priority) - Creates overlays linking each input file to its origin element/path - Generates artifact_overlays for downstream dependency tracing Instantiator (instantiator.py): - Fetches base actions from CAS and resolves SOURCE/ARTIFACT overlays - Replaces file digests in action input trees recursively - Stores adapted actions back to CAS - Optimization: skips overlays for unchanged dependencies Config: - scheduler.speculative-actions flag (default false) in userconfig.yaml Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Synchronize the Remote Execution API proto with the version in buildbox. This adds the subactions field to ActionResult (field 99) for tracking nested executions (e.g. compiler invocations via recc). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Scheduler queues: - SpeculativeActionGenerationQueue: runs after BuildQueue, extracts subaction digests, generates overlays, stores SA by weak key - SpeculativeCachePrimingQueue: runs after PullQueue, retrieves stored SA, instantiates with current dep digests, submits Execute to buildbox-casd's local execution scheduler for verified caching Pipeline wiring: - _stream.py: conditionally adds queues when speculative-actions enabled - _context.py: reads speculative-actions scheduler config flag - element.py: _get_weak_cache_key(), subaction digest storage, _assemble() transfers digests from sandbox after build - _artifactcache.py: store/get_speculative_actions() with weak key path - _cas/cascache.py: fetch_proto()/store_proto() for SA serialization - sandbox.py: accumulates subaction digests across sandbox.run() calls - _sandboxreapi.py: reads action_result.subactions after execution Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

32 tests covering the full speculative actions pipeline without sandbox: Weak key (test_weak_key.py, 13 tests): - Stability: same inputs, dep version changes, dependency ordering - Invalidation: source, command, env, sandbox, plugin, dep changes Generator (test_generator_unit.py, 6 tests): - SOURCE/ARTIFACT overlay production, priority, unknown digests Instantiator (test_instantiator_unit.py, 6 tests): - Digest replacement, nested dirs, multiple overlays, artifact overlays Pipeline integration (test_pipeline_integration.py, 7 tests): - Generate -> store -> retrieve -> instantiate roundtrips - Priming scenario: dep changes, adapted actions have updated digests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Integration tests using recc remote execution through remote-apis-socket (requires --integration and buildbox-run): - test_speculative_actions_generation: autotools build with CC=recc gcc, verifies remote execution and generation queue processed elements - test_speculative_actions_dependency_chain: 3-element chain build - test_speculative_actions_rebuild_with_source_change: patches amhello source, rebuilds, verifies new artifact and generation on rebuild Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add ACTION overlay type to track inter-subaction output dependencies. When a compile subaction produces main.o and the link subaction consumes it, an ACTION overlay records this relationship so that priming can chain the adapted output through to the link action's input tree. Proto: ACTION = 2 in OverlayType, new source_action_digest field (3) to identify the producing subaction by its base action digest. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…lution Generator: accepts ac_service and artifactcache. Processes subactions in order, fetching ActionResults to track outputs. Generates ACTION overlays for intra-element (compile→link) and cross-element (dependency subaction outputs) dependencies. Indexes dependency sources alongside artifacts so that both SOURCE and ARTIFACT overlays are generated for the same file digest, enabling fallback resolution (SOURCE > ARTIFACT > ACTION). Instantiator: accepts ac_service. Resolves overlays with fallback — once a target digest is resolved by a higher-priority overlay, lower-priority overlays for the same target are skipped. Cross-element ACTION overlays fall back to action cache lookup when source_element is set. Generation queue: passes ac_service and artifactcache to generator. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…p callbacks Rearchitect the priming queue to run concurrently with building by using the PENDING state pattern. Elements with stored SpeculativeActions but unbuilt dependencies enter the priming queue as PENDING instead of READY, holding them while background priming runs in the scheduler's thread pool. Element: new _set_build_dep_cached_callback fires each time a build dependency becomes cached (unlike _set_buildable_callback which fires only when ALL deps are cached). Enables incremental overlay resolution as dependencies complete one by one. Priming queue lifecycle: - PENDING: background priming fires immediately via run_in_executor, submitting independent subactions fire-and-forget - Per-dep callback: as each dep completes, re-attempts overlay resolution for newly available ARTIFACT/ACTION overlays - READY (buildable): final pass resolves remaining ACTION overlays (producing subactions now in AC), submits remaining - Done: element proceeds to BuildQueue with all actions primed Unchanged actions (instantiated digest equals base digest) skip submission — already in the action cache from the previous build. The Execute submission reads the first stream response to confirm acceptance by casd, then drops the stream. The action executes asynchronously and its result appears in the action cache. Architecture docs updated for the concurrent priming design, overlay fallback resolution, and data availability considerations. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…solution Replace per-element adapted_digests/action_outputs state with a global _instantiated_actions dict (base_action_hash -> adapted_action_digest) shared across all elements during priming. This fixes cross-element ACTION overlay resolution when dependency elements are adapted but not rebuilt (e.g. intermediate files like generated headers or .o files produced by a dependency's subactions). Key changes: - Generator: fix overlay sort order to SOURCE > ACTION > ARTIFACT. ACTION overlays resolve intermediate files (.o, generated headers) not present in artifacts, so they should be tried before ARTIFACT. - Instantiator: replace action_outputs parameter with global instantiated_actions dict. Add step-0 already-instantiated check. Add resolved_cache parameter to avoid re-resolving overlays across passes when an SA is deferred. - Priming queue: global _instantiated_actions dict and _primed_elements set as class-level shared state. Unresolvable ACTION overlays are removed from the in-memory SA proto when their source_element has finished priming. Cache deserialized spec_actions proto on the element so mutations and resolution caches survive across passes. Add dep-primed callback to trigger incremental priming when a dependency finishes priming (earlier than dep-cached). - Element: add _set_build_dep_primed_callback and _notify_build_deps_primed for the dep-primed notification mechanism. - Architecture doc: add 6 example scenarios, expand ReferencedSAs future optimization, describe global instantiated_actions approach. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace the boolean speculative-actions config with tiered modes that let users control the cost/benefit trade-off: - none: disabled entirely - prime-only: use existing Speculative Actions to prime, don't generate new ones - source-artifact: generate SOURCE and ARTIFACT overlays (no AC calls) - intra-element: also generate intra-element ACTION overlays - full: also generate cross-element ACTION overlays Each mode includes all capabilities of the previous modes. Boolean values (True/False) are accepted for backward compatibility. The mode gates which queues are enabled (priming for all except none, generation for source-artifact and above) and which overlay types the generator produces (ACTION overlay logic skipped in source-artifact mode, cross-element seeding skipped in intra-element mode). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Two optimizations to reduce generation overhead: 1. Return input_digests from _generate_action_overlays() so the caller can reuse them for ACTION overlay matching, instead of re-fetching the Action and re-extracting digests (eliminated duplicate CAS reads per subaction). 2. Collect artifact file entries during _build_digest_cache traversal (_own_artifact_entries) and reuse in _generate_artifact_overlays(), eliminating a redundant full traversal of the element's artifact tree. Cross-element ACTION overlays use eager dependency output seeding because they enable earlier resolution than ARTIFACT overlays (the dep's adapted actions are available via instantiated_actions before the dep's full build completes). The docstring on _seed_dependency_outputs explains this trade-off. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add an in-memory cache for parsed Directory protos to avoid redundant CAS reads during overlay resolution and tree modification. Many overlays reference files in the same directory trees — the same intermediate Directory protos were being read from disk repeatedly. The cache is keyed by digest hash and shared across all subactions within a single instantiator instance (~1MB for ~10K directories). Used in both _find_file_by_path() (overlay resolution) and _replace_digests_in_tree() (action adaptation). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add per-element ActionResult cache to avoid redundant GetActionResult gRPC calls when checking ACTION overlay resolvability across background, incremental, and final priming passes. Once an ActionResult is found for an adapted action digest, it is cached and reused on subsequent passes. Negative results (action submitted but not yet complete) are NOT cached since the result may become available on the next pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Optimize _prefetch_cas_blobs to reduce remote FetchTree latency: 1. Deduplicate input root digests — many subactions share the same input trees, so redundant FetchTree calls are eliminated. 2. Issue FetchTree calls concurrently via ThreadPoolExecutor (up to 16 workers) instead of sequentially. Individual FetchTree calls are preserved (no synthetic root) to maintain remote cache hit rates — input roots from actual builds are likely already cached on the remote. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add 6 tests verifying each mode produces the correct overlay types and makes the expected number of AC calls: - source-artifact mode: no ACTION overlays, zero AC calls - intra-element mode: ACTION overlays for within-element chains only, AC calls limited to own subactions (no dep seeding) - full mode: cross-element ACTION overlays from dep subactions - backward-compat: enum values exist and are distinct - AC call counting: verifies source-artifact makes 0 calls, intra-element makes exactly N calls (one per subaction) Also updated FakeArtifactCache to support lookup by artifact identity (used by _seed_dependency_outputs for cross-element ACTION overlays). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

sstriker · 2026-03-23T21:56:44Z