feat(pixelflow-search): measure rule-order effect on real shader/glyph kernels by jppittman · Pull Request #1101 · jppittman/pixelflow

jppittman · 2026-09-02T06:41:15Z

Summary

Round 2 v3 found the base-62 sweep order dominant on synthetic classical expressions (96.58% regret at B=100 under production order). JP: "the synthetic small kernels are not the workload; shaders and the real glyph/cell-grid kernels are" — this re-runs the same comparison on 204 real kernels: 12 shader_bench ShaderToy ports, the psychedelic shader kernel, one 623-node packed cell-grid geometry, and the 95 ASCII glyph arenas at both densities.
Order still matters, far less than on synthetics: production order carries 21.8% median anytime regret at B=100, falling to 5.1% by B=12800. The numeric-first static reorder is best-or-tied at every checkpoint and reaches 0% by B=12800. In the exact production regime, switching to numeric-first order changes what ships on 167/204 kernels (146 improved / 21 worse), median cost ratio 0.9702.
Full writeup: docs/results/2026-09-01-rule-order-real-kernels.md (+ .csv/.json, 1,020 rows).

Test plan

cargo test -p pixelflow-search — 197 passed, includes the new rule_order module tests (NUMERIC_FIRST_ORDER pinned + re-derived from the frozen report) and the saturation_stop test suite.
cargo test -p pixelflow-core — cell-grid CellGridGeometry fix (frame_w/frame_h) verified.
cargo build --workspace --tests — clean.
cargo check -p pixelflow-ir --no-default-features — no_std still clean.
cargo clippy -p pixelflow-search -p pixelflow-pipeline -p pixelflow-core --tests — clean.
cargo fmt applied.
Measurement harness run: PIXELFLOW_TELEMETRY_DIR=<dumps> PIXELFLOW_RULE_ORDER_INCLUDE_D2=1 cargo test -p pixelflow-search --release -- --ignored rule_order_real_kernels --nocapture --test-threads=1 — 204 kernels × 5 arms, 1,020 rows, 189s.

🤖 Generated with Claude Code

chatgpt-codex-connector · 2026-09-02T06:41:20Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

github-actions · 2026-09-02T06:41:25Z

🤖 Hi @jppittman, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

…inds, not iteration cap optimize_runtime_arena_uncached computes a SaturationResult and discards it (runtime.rs:128-133), so production couldn't say whether a real core-term kernel quiesces, hits the class cap, hits the iteration cap, or hits the 200ms wall clock. Adds three #[ignore]d measurement tests — no public API change, no production behavior change: - pixelflow-core: dump the real packed cell-grid arena at core-term's startup/resize geometries (cell_grid.rs) - pixelflow-graphics: dump the real per-glyph arenas via Font::glyph_kernel_scaled for all 95 printable-ASCII chars at both bake densities (new tests/production_glyph_arena_dump.rs) - pixelflow-search: replay optimize_runtime_arena_uncached's exact calls on every dumped arena, plus two unceilinged references (4x iterations, and 4x iterations + 4x class cap) to distinguish class-cap-bound from genuinely-converged trajectories (runtime.rs) - pixelflow-compiler: measure the winding kernel's separate macro-time Dwrt e-graph (ir_bridge.rs) — finding it never runs for this kernel at all (BitAnd has no e-graph Op, so differentiate_in_optimizer's representable guard bails before constructing an EGraph) Result (173/193 kernels measured; the remaining 20 are density-2.0 duplicates of already-measured characters, see the doc for why the run didn't finish): only 11/173 (6.4%) quiesce under production's budget; 117/173 (67.6%) are class-cap-bound; 45/173 (26.0%) hit the 200ms wall clock (inflated by exceptional machine contention during this run — flagged per-row as machine_dependent). Truncation cost is median 0%, p90 ~16-19%, worst case 47.2%. The cell-grid kernel — compiled at every core-term startup and resize — is class-cap-bound but shows 0% loss. 23/173 rows show more saturation extracting worse code under the static cost model, a real caveat for "more budget is strictly better." Full writeup, per-kernel CSV, and an integrity note (a mid-run attempt by another agent to inject a production API change was reverted) in docs/results/2026-09-01-production-saturation-telemetry.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

`EGraph::saturate_with_limits` has four ways to return — converged, iteration cap, class cap, wall clock — and reported none of them: `SaturationStats` carried only `iterations` and `total_unions`, and `SaturationResult::saturated` reads `iterations < max || total_unions == 0`, which calls a class-cap or timeout stop "saturated". Every consumer that needed the reason (guide_headroom.rs, the production-telemetry harness in runtime.rs) has been inferring it from counts or from extra reference runs. Adds `SaturationStopReason { Converged, IterationLimit, ClassLimit, Timeout }` and a `stop_reason` field on `SaturationStats` and `SaturationResult`, set at each break site of the one loop that decides when to stop. No behavior change; `saturated` is kept as is. This is exactly the graph.rs / mod.rs / saturate.rs hunk set of origin/claude/saturation-telemetry-flag cd9dcdf (identical in 586d9cf), applied as a git patch of those three files; the feature-flagged JSONL telemetry in that commit is not brought in. `SaturationStats` loses its `Default` derive (no default stop reason is honest); nothing used it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… not from reference runs The telemetry harness classified production's stop by inference: two extra unceilinged runs per kernel, compared by iteration count and trajectory signature (`classify` / `classify_reference` / `classify_lifted`). That is exactly the "infer quiescence from counts" mistake the typed `SaturationStopReason` exists to close. The harness now records `SaturationResult::stop_reason` for the production run and for both generous runs; the three classify functions are gone. The winding-kernel Dwrt test in ir_bridge.rs carried the same inference (`stopped_on_its_own` / `cap_bound`, unreachable in practice because the representable guard bails first) and now reads `SaturationStats::stop_reason` too. The generous runs stay, but only as the truncation-LOSS measurement: `ref` (4x iterations, same class cap, no clock) isolates the 200ms wall clock's bite; `lifted` (4x iterations, 4x class cap, no clock) is the whole budget's. Whether lifting the cap changed the trajectory is kept as a plain column (`cap_lift_changed`), a measured fact rather than a classification. Also: - a per-KERNEL wall-clock ceiling (`PIXELFLOW_TELEMETRY_KERNEL_CEILING_S`, default 1200s) shared by the two generous runs, enforced by the loop's own deadline and reported: a cut generous run shows `Timeout`, the row's loss against it is `NA`, and the row is listed under "loss unmeasured" — never skipped, never a panic mid-table; - the 0/0 loss of the 1-node space glyph is `NA`, excluded from quartiles (the old harness would have produced NaN and panicked in `median`); - host load averages recorded at start and end (`<out>.meta`) because the wall-clock stop is the one verdict that depends on the host; - the deterministic-replay check is now stated against the type: when production stopped `Converged` or `ClassLimit` (no clock involved), the same-cap generous run must retrace it exactly, or the row is fatal; - `cargo fmt` applied to the three telemetry test files the measurement commit left unformatted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…scence apply_rule_at_index_timed truncates its own scan at the class budget or deadline and keeps classes.len() at or under max_classes, so the outer `classes.len() > max_classes` check in saturate_with_limits essentially never fires: a class-capped run ends with a zero-union sweep and was labeled Converged. (The production cell-grid kernel settles at 4,768 classes under the 5,000 cap this way.) ApplyResult now carries `truncated`, and the loop reads it: a zero-union sweep is Quiesced only if every rule scanned to completion; otherwise it is Timeout (deadline passed) or ClassCap. Read off the loop, not inferred from counts. Also converges the type onto the names the in-flight Phase 3 branch independently chose for the same thing — SaturationStop { Quiesced, ClassCap, IterationCeiling, Timeout }, field `stop` — so its rebase is a duplicate-delete rather than a rename. Pinned by tests/saturation_stop.rs: the busy expression hits ClassCap at production's 10,000-class budget, x + y is Quiesced, 0 iterations is IterationCeiling, Duration::ZERO is Timeout. This is origin/claude/saturation-telemetry-flag faeaffc restricted to graph.rs / mod.rs / saturate.rs / tests/saturation_stop.rs (its telemetry.rs hunk belongs to the feature flag, which this branch does not carry). It replaces an equivalent fix this branch had written independently (`ApplyResult::stop: ScanStop`), dropped in favour of the reviewed one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

c96e0b04 converged the stop-reason type onto the Phase 3 names (`SaturationStop { Quiesced, ClassCap, IterationCeiling, Timeout }`, field `stop`) but left runtime.rs and ir_bridge.rs on the earlier `SaturationStopReason` / `stop_reason` spelling, so the crate's tests did not compile. Rename only; no logic change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…lder RuleOrder::{Production,NumericFirst,Shuffled(seed)} plus build_rule_set, the base-62 sweep-order arms docs/results/2026-09-01-rule-order-real-kernels.md compares. NUMERIC_FIRST_ORDER is pinned from docs/results/2026-09-01-train-guide-report.md (copied in here as the frozen source) and re-derived by a test so a transcription error fails loudly instead of drifting from the report. Also drops the now-dead ExtractionPolicy::Static match in runtime.rs's production-telemetry harness — env_extraction_policy() is unconditionally the static latency prior since the NNUE extraction-head arm was deleted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…dget-stop saturation Phase 3 recalibration infrastructure (docs/plans/2026-08-31-guide-design-revision.md SS2.3/SS5), continuing the interrupted 2026-09-01 round: - egraph/anytime.rs: run_anytime_curve + APP_CHECKPOINT_GRID — the ONE definition of the anytime metric (cost under CostModel::latency_prior() vs cumulative rule applications, geometric grid 25..204800). Wall-clock exists only as a per-curve safety ceiling that panics if it binds. - EGraph::saturate_until_applications with an explicit SaturationStop reason (quiesced / app-budget / class-cap / sweep-ceiling / timeout) — budgets denominated in rule applications, never sweeps or wall-clock; mid-sweep budget/cap stops can never be mislabeled as quiescence. - provenance.rs: application_id recorded on UnionEvent (zero-cost, from the already-active application context), derivation_ancestors_tight — the tightening variant (subset of the loose labeler bound, superset of the strict node-on-path bound), plus a read-only origins() iterator. - labeler.rs: EpisodeLabels::compute_tight and compute_strict sharing the from_load_bearing aggregation tail with the existing loose compute. - examples/oracle_filtered_budget_curves.rs: converted to the shared anytime loop; checkpoints now in applications (the 2026-08-30 run was null because 97.8% of expressions ended before the old fraction grid's first sample); shaders-in-sample check compares at the final grid target; class cap is the fixed production-tier environment cap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…dget-stop saturation Cherry-picked from claude/phase3-round2 (e9517bd): egraph/anytime.rs (run_anytime_curve + APP_CHECKPOINT_GRID, the one anytime-metric definition docs/results/2026-09-01-rule-order-real-kernels.md reuses for the anytime table) and EGraph::saturate_until_applications with an application-budget stop reason. Merged its SaturationStop enum (adds ApplicationBudget) with the one already on this branch (from the saturation-telemetry cherry-picks) into a single five-variant enum instead of carrying two independently-derived types with the same name. Dropped the unrelated candidate.rs wiring (CandidateFeatures/CandidateKey/ClassContentKey/Firing, guided-saturation research infra) that commit's mod.rs diff referenced but that this branch has no other use for and does not include. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…h kernels docs/results/2026-09-01-rule-order-real-kernels.{md,csv,json}: replays the exact production optimize_runtime_arena_uncached call (config_for_node_count tier, saturate_with_full_budget, static-latency-prior extraction) and the application-denominated anytime curve (B in 100..12800) under five rule-sweep orders -- production all_rules(), the pinned numeric-first static reorder, and three seeded shuffles -- on 204 real kernels: the 12 shader_bench ShaderToy ports, a hand-transcribed psychedelic shader kernel, one 623-node packed cell-grid geometry, and the 95 ASCII glyph arenas at both display densities. Answers JP's question from Round 2 v3 (docs/plans/2026-09-01-phase3-round2-registration-v3.md): order still matters on real kernels but far less than the 96.58%-regret synthetic finding -- production order carries 21.8% median anytime regret at B=100, falling to 5.1% by B=12800; numeric-first is best-or-tied at every checkpoint and reaches 0% by B=12800. In the production regime itself, switching to numeric-first changes what ships on 167/204 kernels (146 improved / 21 worse), median cost ratio 0.9702. Harness pieces: - pixelflow-search/src/egraph/rule_order.rs: RuleOrder, NUMERIC_FIRST_ORDER (pinned from docs/results/2026-09-01-train-guide-report.md, copied in here, and re-derived by a test), build_rule_set. - pixelflow-search/src/egraph/anytime.rs, EGraph::saturate_until_applications: cherry-picked from claude/phase3-round2 (e9517bd), merged with this branch's own SaturationStop (from the saturation-telemetry cherry-picks) into one five-variant enum instead of two independently-derived types with the same name. - pixelflow-search/src/runtime.rs: run_with_rules (production regime, parameterized over the rule set) and anytime_curve_arena (the anytime loop re-driven through the private arena_to_egraph constructor, since real glyph arenas contain runtime-only mask/int ops the public EGraph::add_arena panics on), both using the materialized extracted arena's real cost (arena_cost) rather than ExtractedDAG::total_cost's cycle-penalty-inflated DP total -- an early draft used the DP total directly and produced regret numbers in the billions of percent before this was caught. - pixelflow-pipeline/tests/shader_and_psychedelic_arena_dump.rs: mints the 13 shader/psychedelic arena dumps this measurement needed, in the same text format the existing glyph/cell-grid dumpers use. - pixelflow-core/src/lattice/cell_grid.rs: fixes the cell-grid telemetry dumper for CellGridGeometry's frame_w/frame_h fields (added on main since the saturation-telemetry branch this was cherry-picked from). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…H-a not supported Read-only offline probe, no fix: after production saturation (config_for_node_count + saturate_with_full_budget, exactly as optimize_runtime_arena_uncached calls them), clone the e-graph and run a full upward-congruence-closure sweep to fixpoint on the clone, then compare. Corpus: 206 real kernels (12 shader_bench ports, 1 psychedelic shader, 3 packed cell-grid geometries at the sizes core-term compiles, 190 glyph arenas) reused from claude/rule-order-numeric-first's dumpers (cherry-picked onto this branch: pixelflow-core/src/lattice/cell_grid.rs's dump test, pixelflow-graphics/tests/production_glyph_arena_dump.rs, pixelflow-pipeline/tests/shader_and_psychedelic_arena_dump.rs), plus 200 size-stratified synthetic classical expressions from BwdGenerator. Result: closure finds 922 additional unions total (0.24% of pooled live classes; median per-kernel effect is exactly 0%, p90 0.49%). It never regresses extraction cost and occasionally improves it (median -3.96% on the 23/406 kernels it touches at all) — small punctuation glyphs see up to -11%. But it does not explain the 5,000-class cap binding on most real kernels: only 2/225 cap-hit kernels have a live (canonical) class count that even reaches the cap in the first place, and closure rescues neither of those 2 — the other 223 were already, on median, less than a third the cap's size in live terms before any closure ran. The raw-vs-live gap the cap actually binds on is dominated by genuine e-graph growth from rule application, not by unrecognized congruence. H-a (under-merging inflates class count so the cap binds early) is not supported by this measurement. H-b (cheap, n=4 real kernels): rule order does change the missing-congruence count materially on 2/4 samples, but in the wrong direction to explain MORE congruence on the table on both kernels where it differs, yet #1101 found numeric-first best-or-tied on cost. pixelflow-search/src/egraph/rule_order.rs (RuleOrder, NUMERIC_FIRST_ORDER, build_rule_set) and docs/results/2026-09-01-train-guide-report.md (the pinned order's source-of-truth table, needed by rule_order's own reproducibility test) are cherry-picked from origin/claude/rule-order-numeric-first verbatim — that branch has diverged too far from main (unrelated ir_bridge/optimize.rs rewrites) to merge wholesale, so only the additive, self-contained pieces this probe needs came across; nothing in graph.rs/saturate.rs/all_rules() ordering changed. Probe: pixelflow-search/src/runtime.rs, mod congruence_gap_probe, docs/results/2026-09-02-missing-congruence.{md,csv,json}. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… H-a not supported (#1110) ## Summary Read-only measurement for #1106, no fix. THE FINDING: `EGraph::union(x, y)` pushes only the merged class onto the rebuild worklist; nothing walks the e-nodes elsewhere in the graph that reference `x`/`y` as a child (no e-node parent list exists), so `Add(x, z)`/`Add(y, z)` merge only incidentally. This PR quantifies how much congruence that misses, and whether it explains the 5,000-class cap binding on most real kernels (H-a). **Method**: for each kernel, run the exact production regime (`config_for_node_count` + `saturate_with_full_budget`, as `optimize_runtime_arena_uncached` calls them), then clone the post-saturation e-graph and run a full upward-congruence-closure sweep to fixpoint on the clone. Compare. **Corpus**: 206 real kernels (12 shader_bench ports, psychedelic shader, 3 packed cell-grid geometries, 190 glyph arenas — dumpers cherry-picked from `claude/rule-order-numeric-first`, which has diverged too far from `main` to merge wholesale) + 200 size-stratified synthetic classical expressions from `BwdGenerator`. ## The five numbers 1. **922** additional unions found by closure, pooled — **0.24%** of pooled live classes. 2. **Median per-kernel class-count reduction: 0.00%** (p90 0.49%, max 10.5%). 3. **Median extracted-cost change: 0.00%** — never a regression; on the 23/406 kernels it touches at all, median **-3.96%** (up to -11% on small punctuation glyphs). 4. **Cap correction (the number that matters)**: of 225 cap-hit kernels, only **2** have a live class count that even reaches the 5,000 cap in the first place — the other 223 were already, on median, under a third the cap's size in *live* terms before any closure ran. Closure rescues **0 of the 2** that do reach it. 5. ClassCap-stopped vs. not: **no difference** — both show median 0% reduction, 0% cost change. ## Verdict **H-a is not supported.** The offline upward-closure gap is real but small (922 unions / 0.24% pooled) and does not explain why the 5,000-class cap binds on most real kernels — that cap trips on genuine e-graph growth (rewriting fan-out), not on unrecognized duplicates. Upward-merging is not worth implementing as a fix for the cap story; it may still be worth a narrow, targeted fix purely for the ~5-11% extraction-cost wins it recovers on small kernels, which is a much smaller case than "explains the cap." H-b (cheap, n=4): rule order does change the missing-congruence count materially on 2/4 sampled kernels, but in the direction that argues *against* reframing #1101/#1088 as a congruence-completeness artifact — numeric-first leaves *more* congruence on the table on both kernels where it differs, yet was found best-or-tied on cost in #1101. Full writeup, per-kernel data, and every table: `docs/results/2026-09-02-missing-congruence.{md,csv,json}`. ## Test plan - [x] `cargo build --workspace` - [x] `cargo test --workspace` (all green) - [x] `cargo clippy --workspace --all-targets -- -D warnings` (clean) - [x] `cargo fmt --all -- --check` (clean) - [x] `cargo check -p pixelflow-ir --no-default-features` - [x] `cargo check -p pixelflow-search --no-default-features` - [x] Probe itself run: `PIXELFLOW_CONGRUENCE_ARENA_DIR=<dir> cargo test -p pixelflow-search --release --lib -- --ignored missing_congruence_measurement` No production code path (`union`, `rebuild_budgeted`, `all_rules()` order) changed — this PR is measurement-only, so it does not need the A/B-table-or-DRAFT gate (nothing ships differently for any kernel). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: JP Pittman <jppittman@jpptech.dev> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

… each Working them empirically rather than by inspection changed the picture. The blocker is never the stop-reason redesign by itself — main's landed version strictly supersedes every branch's, including SaturationStop::ApplicationBudget and Budget::Applications. What blocks each branch is whatever else it added to the same files, and that splits three ways. Kind 1 (superseded core, mechanical): #1109 and #1087, both now reconciled, verified and pushed. Records the recipe and two traps worth reusing — git interleaves the two harness modules inside each other's bodies and produces an unclosed delimiter, so take main's file whole and re-append; and each branch's saturation_stop.rs tests are a strict subset of main's, so nothing is lost. Kind 2 (#1101, #1103): a 237-line core makes these look mechanical, but each threads ApplicationId through UnionEvent, ApplicationRecord and derivation_ancestors_tight across three files that main rewrote. Records both traps hit while attempting #1101: checkout --theirs destroys the additive work, and hunk-by-hunk leaves saturate_until_applications fragments orphaned inside main's rewritten function. Kind 3 (#1084, #1088, #1091, #1095, #1096): 800-1,100 lines of GuidedSaturation per branch. Taking main's would delete the experiment. Five independent lines, no shared resolution. Also flags that #1087 and #1101 both add a module named production_telemetry; whichever lands second needs a rename. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NV5ZT2mxTnBC66unvBE7FF

Records the structural finding this day's work produced, which matters more than any individual merge: reconciling a branch against the saturation seam has a half-life measured in hours. Evidence. The board went 5/15 clean to 7/15 as #1054, #1109 and #1087 were fixed, then back to 5/14 the moment #1087 merged — its landing re-conflicted #1109 and #1114, both clean an hour earlier. #1109 needed reconciling twice in one day for the same end-of-file append collision. #1114 was clean at 10:10 and needs an API port by 11:00 (Budget gained a field, SaturationStop::ClassCap changed arity, reused helpers went private). And the #1087/#1101 collision this document predicted materialized exactly as described. One mechanism behind all of it: egraph/graph.rs, egraph/saturate.rs and runtime.rs are one hot seam with ten branches queued on it, and each landing invalidates the rest. The fix is not more reconciliation — it is to rebase the queue as a batch after each seam landing, or freeze the seam until it drains. Corollary: reconcile in landing order and land promptly, because a reconciled-but-unlanded branch is a wasting asset. #1087 is the counter-example in the good direction, reconciled and merged the same hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NV5ZT2mxTnBC66unvBE7FF

An earlier draft called these branches '800-1,100 lines to re-apply', conflating lines the branch adds with lines in conflict. #1091 adds ~937 lines to graph.rs/saturate.rs but most auto-merge: the actual conflict is 12 hunks, ~395 lines, most resolving mechanically to main's landed design like Kind 1. Worked #1091 to the bottom, so the mechanical parts are now written down: the superseded saturate_until_applications (154 lines) drops since saturate_bounded takes max_applications and SaturationStop already has ApplicationBudget; the saturate.rs/mod.rs hunks are import merges; AppBudgetSaturationStats is the branch's own type and re-homes into saturate.rs; and Cargo.toml auto-merges into two [features] blocks, which cargo reports against core-term rather than this crate, making it look unrelated. What actually blocks it is a design collision in runtime.rs. ProductionSaturation, saturate_for_production and production_saturation_probe exist only on these branches. The branch factored production's saturation into a shared seam so its probe provably runs production's code rather than a drifting copy; main rewrote the same function around Optimizer without that seam. Reconciling means re-deriving the seam and re-pointing the probe — authoring, not merging, and a subtle error breaks the property the probe exists to give. Also updates the #1087/#1101 collision from predicted to happened, with the resolution: main's docs are the later run and win, #1101's module is the superset and grafts four named functions onto main's already-ported copy — taking #1101's wholesale would silently revert that port. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NV5ZT2mxTnBC66unvBE7FF

…arately Explains why this seam is contested rather than merely busy. #1083 (landed), #1109, #1114 and #1101 are each answering the same design question — what it means for saturation to stop at the class cap — and answering it differently: #1083 ends the run on it; #1109 separates classification from termination and measures the break costing a more expensive arena on 140 of 204 real kernels; #1114 says it should name which ceiling bound, Live or Allocated; and #1101's saturate_until_applications reported Quiesced where main's saturate_bounded reports ClassCap for the same run. That last one surfaced as a test failure, not a merge defect. This pass ported #1101's harness onto Optimizer/Budget::Explicit mechanically — every rename resolved, workspace built — and its own clamped_rows_freeze_the_final_state test then failed with left: ClassCap, right: Quiesced. The test is the branch's own and passes on the branch, so it is detecting a real behavioural difference between the entry point it was written against and the one that replaced it. Consequence: #1101 cannot be reconciled without deciding whose semantics the measurement runs under, and that decision is shared with #1109 and #1114. Answering it once unblocks three branches; answering it per-branch guarantees the results disagree. Nothing pushed to #1101 pending it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NV5ZT2mxTnBC66unvBE7FF

Measured what actually depends on the deleted saturate_until_applications entry point: all seven conflicted saturation branches do, at 6 to 27 call sites each, plus 3 to 7 files each using run_anytime_curve. main replaced it with Budget::Applications, and the two disagree on when a class cap ends a run — which #1101's own clamped_rows_freeze_the_final_state test detects. So the conflict count is misleading. This is not eight independent reconciliations; it is one unresolved design decision with seven branches queued behind it, plus #1072, which is unrelated and structural. Porting the difference per-branch is how seven registered experiments end up measured under seven slightly different stopping rules. Answering it once — does an application-budgeted anytime run stop at the class cap, or record it and continue? — converts all seven from needing judgement into the mechanical recipe already written down here, which this pass executed end to end three times (#1087, #1109, #1114). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NV5ZT2mxTnBC66unvBE7FF

…ct reference `extract_dag` is not argmin under its own cost model three ways over (issue #1111): it sums a TREE cost so sharing is never priced, it is a single DFS rather than a fixpoint so a class whose child is `on_stack` is scored `CYCLE_COST` and never revisited, and `total_cost` is read (extract.rs:1636) before `repair_choices_well_founded` may change the choices (:1639). That is why every budget-loosening change (#1101, #1109, #1114) improves most kernels and regresses a minority: a larger e-graph is not monotonically better under a lossy chooser, even though the exact optimum over it can only fall. This measures the gap. Two references, both under `CostModel::latency_prior()` and `Optimizer::production()`, so the comparison is chooser-vs-chooser: - Knuth's algorithm (Dijkstra generalized to AND-OR graphs) — the EXACT minimum tree cost, in polynomial time, on every kernel. That is the objective the DP names, so it isolates the DP failing its own objective from the objective being the wrong one. - A branch-and-bound over per-class choices for the true DAG optimum, with dominance filtering, a chain lower bound, and reduced-cost fixing. NP-hard, so it is budgeted and reports UNSOLVED rather than a truncated answer; a brute-force enumeration cross-check pins it on every instance small enough to enumerate. 302 kernels (206 real arena dumps + a synthetic size ladder). Headline: the DP misses its own tree optimum by p90 3.26% and worst 26.79%, attaining it on 230/302; the exact DAG optimum closes on 89 and greedy is optimal on 86 of those. `total_cost` fails to describe the returned term on 132/302. And on glyph16/32:U+004B — one of #1114's regressions — the roomier e-graph provably CONTAINS a term at 1013 while greedy returns 1132 against 1047 on the smaller graph: an extraction failure, not a budget failure. Measurement only; no production behavior changes. The probe lives in its own file rather than growing runtime.rs (2,631 lines and flagged for size), and the arena-dump loader both probes share moves to `arena_corpus`. Results: docs/results/2026-09-02-extraction-gap.{md,csv,json} Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…trades The measurement landed in the previous commit; this is what a sequencing decision needs from it, written into the same doc. Headline, with its caveats attached: the greedy DP leaves **0.258% pooled / 14.557% worst case** against the TREE optimum — the objective `extract_dag` itself names — over all 302 kernels with nothing excluded, and that is a floor. The true DAG optimum is lower but NP-hard: it closed on 89 of 302, and exactness stops at ~100-200 reachable classes while a production glyph saturates to a median 1,755. The two references also describe different populations — all 68 kernels the tree optimum beats are real (64 glyph, 4 shader), zero synthetic — so no single averaged ratio is honest. Mechanism split, and what each fix buys: - (i) single DFS, not a fixpoint — 5% of the proved loss, but **0.258% pooled over all 302 with no time limit, 68 kernels improved, 14.557% worst**, and it is the cheapest of the three. FIRST. - (ii) tree cost instead of DAG cost — 95% of the proved loss, but that is a small-kernel statement: beyond the tree optimum the entire certified headroom on this corpus is **0.028% pooled**. A research direction, not a fix. - (iii) `total_cost` read before `repair_choices_well_founded` — recovers no cycles at all, and is worse than "stale": of the 132/302 mismatches **not one** is ordinary-magnitude. 92 report exactly `usize::MAX` (the `.unwrap_or` at extract.rs:1636, taken when the DFS never resolves the root) and 40 report 1x, 2x or 3x `usize::MAX / 4`, the `Dwrt` sentinel from cost.rs:292. No shipped result is corrupted by it — #1101/#1109/#1114's harnesses re-score the arena (`arena_static_cost`) and `guide_headroom.rs:395`'s corpus holds no sentinel — so it is a live trap, not a live wound. Free to fix. Sequencing: fix extraction before taking #1101, #1109 or #1114. The two `U+004B` regressions in #1114 are extraction failures, not budget failures — the roomier e-graph provably contains a term at 1013 while greedy returns 1132 — so the fixpoint turns them into improvements, which no budget tuning can do. `psychedelic` stays unresolved rather than exonerated. Cost: extraction runs ONCE per compile; saturation runs 8,729,067 rule applications over 800 expressions in the guide-headroom corpus, and #1114 doubles that for +2.03%. A log factor on the once-per-compile pass is the cheap place to spend. Cross-links #1111, §1.L5 of docs/plans/2026-09-02-optimizer-api.md, #1101, #1109, #1114. Docs only; no code touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… exact (#1115) Three independent budget-loosening changes each improved most real kernels and regressed a minority — #1101 (numeric-first rule order, 146 up / 21 down), #1109 (removing the class-cap break, 140/16), #1114 (live-counted class budget, 155/4, +2.03% for 2.00x applications). The mechanism named in #1114 and established in #1111 is that `extract_dag` is a greedy DP over a static prior and is **not argmin** (§1.L5 of `docs/plans/2026-09-02-optimizer-api.md`), so a larger e-graph is not monotonically better even though the *optimum* over it can only fall. This PR answers the question that sequences those trades: **how much is the greedy chooser leaving on the table?** Measurement + docs only — no production behavior changes. ## Method Two references, both under `Optimizer::production()` and `CostModel::latency_prior()` — the same cost function the DP uses — so the comparison is chooser-vs-chooser, never cost-model-vs-cost-model. 302 kernels: 206 real `.arena` dumps + a 96-point synthetic size ladder. 1. **Knuth's algorithm** (Dijkstra generalized to AND-OR graphs) — the **exact minimum tree cost, in polynomial time, on every kernel**, no time limit. That is the objective `extract_dag` *names*, so it separates "the DP fails its own objective" from "the objective is the wrong one". 2. **Branch and bound** for the true DAG optimum (each chosen class priced once) with dominance filtering, an admissible chain lower bound, and reduced-cost fixing. NP-hard, so it is budgeted and reports `UNSOLVED` rather than a truncated answer; cross-checked against brute-force enumeration on every instance small enough to enumerate. ## Headline | reference | scope | pooled recovery vs greedy | per-kernel | |---|---|---|---| | **tree optimum** (exact, polynomial, no limit) | **all 302, nothing excluded** | **0.258%** (real-only 0.284%; shaders 2.62%) | median 0.000%, p90 0.403%, worst **14.557%**; 68 kernels strictly improved | | **DAG optimum** (4s cap) | 89 of 302 proved | 0.228% on those 89 | greedy already exactly optimal on **86 of 89**; worst ratio 1.1212 | | **certified floor** (cheapest term the search exhibited) | all 302 | 0.286% | strictly beats greedy on **81** | Caveats that travel with the number: it is against the *tree* optimum; exact DAG extraction stops being computable at ~100–200 reachable e-classes while a production glyph saturates to a median **1,755**, so 213/302 are UNSOLVED and excluded from the exact statistic; and all 68 kernels the tree optimum beats are **real** (64 glyph, 4 shader) — zero synthetic — so the two references describe different populations. ## Mechanism split — which of the three to fix | # | defect (#1111 / §1.L5) | share of proved loss | what fixing it recovers | cost | |---|---|---|---|---| | (i) | single DFS, not a fixpoint (`CYCLE_COST`, never revisited) | 10/195 (5%) | **0.258% pooled, all 302, p90 0.403%, worst 14.557%, 68 kernels** — and both `U+004B` regressions | small | | (ii) | tree cost, not DAG cost (sharing unpriced) | 185/195 (95%) | **0.028% pooled beyond the tree optimum** | NP-hard | | (iii) | `total_cost` read at `extract.rs:1636` before repair at `:1639` | 0 | no cycles at all | trivial | The 95/5 split is a **small-kernel statement** — it is computed only where the branch and bound closed. Over the full corpus the picture inverts, and (i) is both the larger measurable recovery and the cheaper fix. Cycle-priced classes are not rare: 238/302 kernels have at least one (median 10, max 284), and on 203/302 the best term the search holds *uses* one. (iii) is worse than "stale": of the 132/302 mismatches **not one** is an ordinary-magnitude discrepancy. 92 report exactly `usize::MAX` (the `.unwrap_or` at `:1636`, taken when the DFS never resolves the root class) and 40 report 1x, 2x or 3x `usize::MAX / 4` — the `Dwrt` sentinel from `cost.rs:292`, summed into a number typed as a cost. **No shipped result is corrupted by it** — #1101/#1109/#1114's harnesses re-score the returned arena (`arena_static_cost`, `runtime.rs:1278`) and `guide_headroom.rs:395`'s 800-expression corpus holds no sentinel — so it is a live trap, not a live wound. ## The finding that settles the sequencing On **`glyph16:U+004B` and `glyph32:U+004B`** — two of #1114's four regressions — the roomier e-graph **provably contains a term costing 1013** while greedy returns **1132** on it and **1047** on the smaller production graph. The graph got strictly better and the chooser gave up 8.1% anyway: **those regressions are extraction failures, not budget failures.** `shader:julia_set` leaves **5.6%** on the table at the production budget with no budget change in play. `psychedelic` (766 → 816) exhibited no cheaper term inside the limit — **unresolved, not exonerated**. ## Sequencing **Fix extraction before taking #1101, #1109 or #1114.** The fixpoint is free at run time, recovers more on real kernels than #1114 buys for 2x the rule applications where they overlap, and turns #1114's `U+004B` regressions into improvements — which no budget tuning can do. Cost of a fixed extractor (estimate, not benchmarked): Knuth is `O(Σ arity · log V)` against the DFS's `O(Σ arity)` — a small constant and a log, not an order — and it completed on all 302 kernels including the 4,847-class `psychedelic`. **Extraction runs once per compile; saturation runs 8,729,067 rule applications over 800 expressions** in the guide-headroom corpus (median 194.5, p90 31,197, max 996,047), and #1114 doubles that for +2.03%. This is the cheap place to spend. ## Shape - Probe in its own file, `#[cfg(test)]` + `#[ignore]`d — `runtime.rs` was already 2,631 lines and flagged for size. - The shared `.arena` loader moved out of `runtime.rs` into `arena_corpus` (net −124 lines there), so the two probes that read the corpus do not carry two copies of the dumpers' inverse. - No public API added; no production path touched. Results: `docs/results/2026-09-02-extraction-gap.{md,csv,json}`. Gates: build, `cargo test --workspace`, `clippy --workspace --all-targets -D warnings`, `fmt --check`, and both no_std checks (`pixelflow-ir` and `pixelflow-search`, `--no-default-features`) all green locally. Refs #1111, #1101, #1109, #1114. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: JP Pittman <jppittman@jpptech.dev> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…y returns (#1117) `extract_dag_scoped` read `total_cost` from the DP table at `:1636`, let `repair_choices_well_founded` rewrite the choice map at `:1639`, and returned the pre-repair number alongside the post-repair choices at `:1661`. #1115's harness measured the consequence: **the reported cost differs from the true cost of the returned term on 132 of 302 kernels** (`docs/results/2026-09-02-extraction-gap.md`, `reported_matches_returned`). `extract()` had the same shape, and `build_extracted_dag_from_choices` fabricated `total_cost: 0` outright. Correctness fix. **No objective change** — the DP chooses exactly what it chose before, and every `choices` vector is byte-identical. Only the reported numbers change. ## What the fix is A new `cost_of_choices(egraph, root, choices, costs, shape) -> ChoiceCost` costs *the choice map it is handed* — nothing is minimized, no node is reconsidered. A cyclic or incomplete map panics rather than returning a number for a term that cannot be materialized. Both extraction entry points call it after the repair. ## Why two fields, not one Per #1116's nuance: the DP minimizes a **tree** cost (each child summed, sharing never priced) while the emitted kernel pays a **DAG** cost (each distinct chosen class once). One number cannot answer both questions, so the field docs name which is which: | field | meaning | who wants it | |---|---|---| | `ExtractedDAG::total_cost` | **tree** cost of the returned term — the objective the DP optimizes | anything comparing the DP against a reference that minimizes the same thing | | `ExtractedDAG::dag_cost` | **DAG** cost of the returned term — what the emitted kernel pays | *"what will this kernel cost?"* | On `shader:julia_set` those are ~1.4e7 against 716 (a 20,000x sharing ratio). **The objective is unchanged — that is #1116 and JP's call.** This PR only makes the reported number honest about which one it is. `Optimized::cost` carries the pair, so `Optimizer::run` no longer discards the total it computes. ## Reader audit — what each caller actually wants | reader | wants | verdict | |---|---|---| | `oracle_filtered_budget_curves` (regret curves) | cost of the kernel each checkpoint would emit → **DAG** | **fixed** | | `guide_headroom`'s `extracted_cost` column | cost of the extracted kernel → **DAG** | **fixed** | | `dag_to_kernel_code`'s generated bench comment | cost of the benched kernel → **DAG** | **fixed** | | compiler's `optimize_via_model` | nothing — builds `ExtractedDAG` only for let-binding placement | **fixed**: passes `Optimized::cost` instead of fabricating `0` | | `production_telemetry`'s `dp_cost` | the DP's own objective value → **tree**, deliberately | correct; now read from `Optimized::cost.tree` (post-repair) instead of re-running the DP | | `extraction_gap`'s `greedy_reported` | the DP's *reported* number, deliberately, to check it | correct; `reported_matches_returned` is now the standing regression check | | `runtime.rs` / `ir_bridge.rs` `arena_cost` | **DAG**, from the materialized arena | correct; their doc comments no longer blame `CYCLE_COST` inflation, which is gone | | `swap_search_reproduces_extract_dags_choice…` | tree | correct | ## Which published numbers were affected Plainly, by harness: - **Safe.** #1101's rule-order results and `docs/results/2026-09-02-missing-congruence.*` — both re-cost the materialized arena (`arena_cost` / `arena_static_cost`), which was the right workaround and is unaffected. - **Safe.** `docs/results/2026-09-02-extraction-gap.md` (#1115) — its greedy/tree-optimal/exact costs come from `Instance::dag_cost` and `Instance::tree_cost`, computed independently of the reported field. The headline gap numbers stand. Only its `reported_matches_returned` / `reported_delta` columns describe the defect being fixed here, and re-running should now put them at 0/302. - **Safe conclusion, stale column.** `docs/results/2026-09-01-production-saturation-telemetry.csv` — the quality metric is `cost` = `arena_cost`; the raw `dp_cost` column carried the pre-repair total and will change on a re-run. - **Safe conclusion, stale column.** `docs/results/2026-08-30-guide-headroom.{md,json}` — the headline is the load-bearing ratio, which never touched cost; the `extracted_cost` column was a tree cost and is now a DAG cost. - **Not safe.** `docs/results/2026-08-30-oracle-filtered-budget-curves.{md,csv}` — its `cost` and `regret_pct` columns *are* `ExtractedDAG::total_cost`, i.e. pre-repair tree costs used as a stand-in for emitted-kernel cost. Those curves need a re-run before the numbers are quoted again. ## Tests - `reported_cost_follows_the_choice_the_repair_rewrote` — a class holding both `Sin(x)` and a self-referential `Neg`, under a cost function that prices `Sin` above the `CYCLE_COST` sentinel. The DP's minimum is the self-reference; the repair rewrites it to `Sin`. Verified to fail against the old read (`left: 4611686018427387903` vs `right: 9223372036854775807` — `CYCLE_COST` reported for a term costing twice that). - `dag_cost_equals_the_materialized_arenas_cost` — the property every measurement in this repo has been assuming, asserted directly against `choices_to_arena`. - `tree_cost_prices_a_shared_subterm_once_per_use_and_dag_cost_once` — pins that the two fields are genuinely different quantities. - `cost_of_choices_refuses_a_cyclic_choice_map`. ## Gates `cargo build --workspace`, `cargo test --workspace` (exit 0), `cargo clippy --workspace --all-targets -D warnings` (exit 0), `cargo fmt --all --check`, `cargo check -p pixelflow-ir --no-default-features`, `cargo check -p pixelflow-search --no-default-features` — all green. Closes#1111. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

jppittman deployed to ci September 2, 2026 06:41 — with GitHub Actions Active

jppittman enabled auto-merge (squash) September 2, 2026 06:41

jppittman deployed to ci September 2, 2026 06:41 — with GitHub Actions Active

jppittman force-pushed the claude/rule-order-numeric-first branch from 2f4d16a to c941d60Compare September 2, 2026 06:41

jppittman changed the title ~~Measure rule-order effect on real shader/glyph kernels~~feat(pixelflow-search): measure rule-order effect on real shader/glyph kernelsSep 2, 2026

jppittman mentioned this pull request Sep 2, 2026
perf(pixelflow-search): order all_rules() numeric-first — the same 62 rules, ~86x lower regret at small budgets on synthetics; real-kernel table in body #1103
Draft

JP Pittmanand others added 9 commits September 2, 2026 00:03

jppittman force-pushed the claude/rule-order-numeric-first branch from c941d60 to 27b469bCompare September 2, 2026 07:03

This was referenced Sep 2, 2026
feat(pixelflow-search): upward congruence closure behind Optimizer::congruence() — A/B says do not ship #1113
Draft
feat(pixelflow-search): budget saturation against LIVE e-classes, not allocated slots #1114
Draft

jppittman mentioned this pull request Sep 2, 2026
docs(results): how much extraction leaves on the table — greedy DP vs exact #1115
Merged

jppittman mentioned this pull request Sep 2, 2026
extract_dag minimizes TREE cost while the real objective is DAG cost — ~95% of the measured extraction gap #1116
Open

jppittman mentioned this pull request Sep 2, 2026
fix(pixelflow-search): report the cost of the term extraction actually returns #1117
Merged

Navigation Menu

feat(pixelflow-search): measure rule-order effect on real shader/glyph kernels - #1101

feat(pixelflow-search): measure rule-order effect on real shader/glyph kernels#1101
jppittman wants to merge 9 commits into
mainfrom
claude/rule-order-numeric-first

jppittman commented Sep 2, 2026

Uh oh!

chatgpt-codex-connectorBot commented Sep 2, 2026

Uh oh!

github-actionsBot commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jppittman commented Sep 2, 2026

Summary

Test plan

Uh oh!

chatgpt-codex-connectorBot commented Sep 2, 2026

Uh oh!

github-actionsBot commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant