feat: Rust crate by arthurpaulino · Pull Request #2 · argumentcomputer/ix

arthurpaulino · 2025-02-04T13:52:08Z

No description provided.

Lays the groundwork for measuring the kernel performance plan (plans/okay-let-s-write-a-lucky-dolphin.md) per audit §10. Counters live in a new kernel::perf module; KEnv carries a PerfCounters field that is dumped from a Drop impl when IX_PERF_COUNTERS=1 is set. Unset is the production default — every increment short-circuits via a LazyLock<bool> so the cost is a single cached branch on the hot path. Wired into the cache get sites the audit identified: - whnf_cache hits/misses (whnf.rs around 201/211) - whnf_no_delta_cache hits/misses (whnf.rs around 437/446) - infer_cache and infer_only_cache hits/misses (infer.rs around 45/51) - def_eq_cache hits/misses (def_eq.rs around 137/149) - def_eq_failure set hits/inserts (def_eq.rs around 360) - per-constant peak/avg MAX_REC_FUEL consumption (recorded in TypeChecker::reset before the next constant resets it) Verified against lake test -- kernel-check-env --ignored: 192156/192156 constants pass in 251s (4% over the 242s baseline; overhead is the LazyLock branch + atomic load in the disabled path). P1: def_rank_id replaces def_weight_id for hint-priority comparison Audit Tier 1 #3 (kernel-perf-adversarial-audit-2026-04-26.md, §4.2): the prior u32 encoding mapped Abbrev to u32::MAX-1 and saturating-added Regular(h) to h+1, which collide at h ≥ u32::MAX-2. When that happens the delta-direction logic treats Abbrev and a maximally-heavy Regular as "same height" and unfolds both, instead of preferring Abbrev as Lean does (compare(d_t->get_hints(), d_s->get_hints()) at type_checker.cpp:910). Replace the u32 weight with a (class: u8, height: u32) tuple compared lexicographically: - Opaque / Theorem / unknown → (0, 0) - Regular(h) → (1, h) (height ordering preserved within class) - Abbrev → (2, 0) (strictly above every Regular) Update the two call sites (is_def_eq lazy-delta height comparison and lazy_delta_step). The map_or default for "missing head" is preserved as (u8::MAX, u32::MAX) — the branch is dead in practice (a_delta && b_delta imply both heads are present) but kept consistent with the prior u32::MAX sentinel. Two regression tests: - def_rank_abbrev_above_saturated_regular: Abbrev outranks Regular(u32::MAX) (the previous saturation collision). - def_rank_regular_orders_by_height: height monotonically orders Regular ranks within the class. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 256s (no regression vs the 242s pre-perf-counters baseline; the +14s is from the IX_PERF_COUNTERS=unset LazyLock branch added in the prior commit, not this change). P2: peel_proj_forall fast-paths syntactic Pi in projection inference Audit Tier 1 #2 (kernel-perf-adversarial-audit-2026-04-26.md, §7.2): infer_proj's two parameter-consuming loops (param peel and field peel) called self.whnf(&r)? unconditionally per iteration, on a body mutated by subst at the previous step. The whnf cache rarely hits between iterations and each call re-traverses the substituted body. Extract a peel_proj_forall(&r, err) helper that: - tries ExprData::All(..) syntactically first (no WHNF call), and - falls back to full self.whnf(e) only when the binder isn't already syntactic Pi. This mirrors Lean's inferProj at type_checker.cpp:582–610. Both projection-inference loops now call peel_proj_forall instead of unconditional whnf. Behaviorally equivalent — same WHNF semantics on miss, no semantic change otherwise. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 258s (parity with the post-pre-work, post-P1 baseline; no measurable regression and the cache-hit-rate counters will move on tactic-heavy workloads under IX_PERF_COUNTERS=1). P3a: WhnfFlags substrate (no behavior change) Lays the foundation for the Lean4Lean architectural alignment described in plans/okay-let-s-write-a-lucky-dolphin.md. Phase 3a is substrate-only — no call site is migrated to cheap mode yet, so behavior is unchanged. Adds: - WhnfFlags { cheap_rec, cheap_proj } with FULL and CHEAP consts and is_full(). CHEAP is currently equal to FULL until Phase 3c wires it. - whnf_core_with_flags (private): the existing whnf_core impl, now threading flags into recursive calls and try_iota_with_flags. - whnf_core / whnf_core_cheap (super): FULL/CHEAP wrappers. - whnf_no_delta_with_flags (private): the existing whnf_no_delta impl with the Prj branch gated on cheap_proj — falls back to full whnf on the projected value when not cheap. - whnf_no_delta (pub) / whnf_no_delta_cheap (super): wrappers. - try_iota_with_flags: gates major-premise WHNF and string-literal constructor reduction on cheap_rec. - try_proj_app_reduce_with_flags: gates projected-value WHNF on cheap_proj. Cache reads/writes (whnf_no_delta_cache, equiv-manager second-chance) are gated on flags.is_full(): cheap callers neither read nor write the cache, preserving the invariant that any cached entry is a fully-reduced normal form. Phase 3b will inline the projection branch into whnf_core to match Lean4Lean's two-layer architecture (refs/lean4lean/Lean4Lean/ TypeChecker.lean:266, 297). Phase 3c will flip CHEAP to enable cheap_proj and migrate specific def-eq sites. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 243s (matches the 242s baseline; substrate adds no measurable overhead when CHEAP == FULL). P3b: inline projection into whnf_core (Lean4Lean architectural alignment) Move the Prj branch from whnf_no_delta_with_flags into whnf_core_with_flags so our whnf_core matches Lean4Lean's whnfCore semantics exactly (refs/lean4lean/Lean4Lean/TypeChecker.lean:284-292, 337-341). Before this commit: whnf_core — beta + zeta + iota + cheap projection (recursive whnf_core on val, no delta) whnf_no_delta — whnf_core + FULL projection (full whnf on val) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta Lean4Lean's architecture has no whnf_no_delta layer. Their whnfCore includes projection, with the cheap_proj flag deciding whether the projected value uses whnfCore (cheap) or whnf (full). After this commit: whnf_core (with WhnfFlags) — beta + zeta + iota + projection (cheap_proj controls val reduction) whnf_no_delta — whnf_core(_, FULL) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta The bare-Prj branch in whnf_no_delta_with_flags is removed — whnf_core now handles it directly. The App-of-Prj branch stays in whnf_no_delta because whnf_core's loop returns once the outermost Prj is resolved; try_proj_app_reduce_with_flags gives one more attempt at the same cheap_proj policy when the outer expression is App(Prj, ...). Pure refactor, no semantic change with CHEAP == FULL. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 270s (within noise of the 243s pre-refactor baseline). Phase 3c will flip CHEAP to enable cheap_proj=true and migrate def-eq's lazy-delta sites surgically per Lean4Lean's pattern. P3c (postponed): document the HeaderParsedSnapshot regression P3a (substrate) and P3b (Lean4Lean architectural alignment) are committed. Phase 3c — flipping CHEAP to enable cheap_proj=true and migrating the def-eq lazy-delta sites — was attempted but reproduced 5 failures on chained projections in Lean.Language.Lean.HeaderParsedSnapshot.* even after P3b inlined the projection branch into whnf_core. The substrate is left in place; CHEAP stays equal to FULL until the regression's root cause is understood. Notes on the regression for the next investigator: - Failures: HeaderParsedSnapshot.{stx,result?,metaSnap,toSnapshot,ictx}, all with 'projection: type mismatch with declared struct'. - The struct `extends` a parent, so each projection is a chained Prj whose val is itself a Prj into the parent. - The error comes from infer.rs's infer_proj at the head-vs-struct_id address compare, after FULL whnf on val_ty. That whnf is FULL, but val_ty was inferred via paths that may have consulted a def-eq cache populated under cheap mode. Possible cache-poisoning suspect: def_eq_cache writes a `false` result for inputs whose lazy-delta loop bottomed out under cheap projections. The cache key uses raw a/b hashes, not cheap-reduced shapes, so a stored `false` is indistinguishable from a FULL `false` by future readers. - Lean4Lean does not have an analogous wide def_eq_cache; their failure cache is keyed only on same-spine pairs in lazyDeltaReductionStep. Future P3c iterations should either prove the cache poisoning theory incorrect or restrict def_eq_cache writes to FULL-derived results.

* Refactor Claim ADT + add merkle infrastructure for ZK aggregation Adds the format hooks needed before recursive verification lands in Aiur. Five-variant Claim ADT with explicit assumption commitments, a canonical Blake3 merkle module, a serializable AssumptionTree for recovering leaf sets, and a Contains claim for the discharge step. Claim ADT (5 variants): - Eval { input, output, assumptions: Option<Address> } - Check { const_addr, assumptions: Option<Address> } - CheckEnv { root, assumptions: Option<Address> } - Reveal { comm, info } -- unchanged, no assumptions - Contains { tree, const_addr } -- new, for inclusion proofs Tag4 reorganized to keep everything in single-byte tags: - 0xE for env, comm, AssumptionTree, and claims (slots 0-7) - 0xF for proofs (slots 0-4; 5-7 reserved) - Comm moved from variant 5 -> 1 Matches the "Variant (0-7)" constraint documented in docs/Ixon.md. Env serialization: - Every .ixe file now carries a canonical merkle root over its consts.keys() in the on-disk header (non-optional, 32 bytes; empty const sets use the zero-address sentinel). - Two envs with the same const set produce byte-identical roots regardless of insertion order. Verified on deserialize. New modules: - src/ix/ixon/merkle.rs + Ix/Merkle.lean: canonical sorted builder, free-form merkle_join composition, membership proofs, domain separation per RFC 6962. - src/ix/ixon/assumption_tree.rs + Ix/AssumptionTree.lean: serializable merkle tree with Leaf/Padding/Node variants. canonical() builds the same shape merkle_root_canonical hashes; join() is O(1) free-form composition. - src/ix/kernel/claim.rs: builders that compute transitive-dep assumptions from an env (build_check_claim, build_eval_claim, build_check_env_claim, env_merkle_root). Other: - Extracted shared BFS walker on Env::bfs_refs + Env::transitive_deps_excl; the inlined test-feature copy in lean_env.rs now calls into it. - Lean FFI export rs_env_merkle_root for cross-impl verification of the env root. - Proof bytes for all variants are uniform opaque ZK bytes; witness data (e.g., Contains merkle paths) is prover-side scratch consumed by the ZK circuit and not transmitted on the wire. - docs/Ixon.md Tag4 tables and env section updated; .ixe extension documented. Recursive verification (the ZK proof generation for Contains and the aggregation discharge transitions) is intentionally deferred to a follow-up. Tests: 993 Rust unit tests pass (was 953 pre-refactor), 813 Lean tests pass with no failures; cargo clippy clean. * Add lazy env deserialization and anon-mode kernel FFI Two related changes to reduce kernel memory + lay groundwork for metadata-isolated typechecking. **Lazy constant deserialization.** `Env::consts` now stores `LazyConstant` (`Arc<[u8]>` + `OnceLock<Arc<Constant>>`) instead of `Constant`. Constants are materialized on first access and the structured form is cached. Sparse access patterns (single-constant typecheck, transitive_deps walk, claim builders) only parse the closure they need; non-reachable constants stay as raw bytes. The `.ixe` section-2 layout gains a Tag0 length sidecar before each constant's Tag4 bytes: old: [addr:32] [Tag4 constant bytes] new: [addr:32] [Tag0 length] [Tag4 constant bytes] The length is section-level framing, not part of the constant's content hash. `Address::hash(raw_bytes) == addr` is preserved. `Constant::put`/`Constant::get` are unchanged. The Lean side (`Ix.Ixon.putEnv`/`getEnv`) reads/writes the new format. **Anon-mode kernel FFI.** New `rs_kernel_check_consts_anon(path, addrs, quiet)` exposes anonymous-mode typechecking by content address. The kernel runs as `KEnv<Anon>` / `TypeChecker<Anon>` with every `M::MField<T>` erased to `()`, so the typechecking logic structurally cannot read metadata. Useful for zkPCC verifiers that hold only addresses. Supporting pieces: - `KernelMode::HAS_META: bool` for future compile-time gating. - `AnonEnv<'a>` wrapper (`src/ix/kernel/anon_env.rs`) exposing only consts/blobs/transitive walks — no `named`/`names`/`comms`. - `Env::get_anon` reads header + blobs + consts, parse-and-drops metadata sections (3-5), returns an `Env` with empty `named`/ `names`/`comms`. Same merkle-root verification as `Env::get`. - `rs_de_env_anon` FFI + `Ix.Ixon.rsDeEnvAnon` Lean wrapper. - `Ix.KernelCheck.rsCheckConstsAnonFFI` Lean binding. Caveat: `ixon_ingress::<Anon>` still consults `Env::named` internally to enumerate work items. The resulting `KEnv<Anon>` is metadata-free so the typechecker is anon, but full ingress-level metadata isolation is a follow-up. Tests: 16 new (9 lazy + sparsity; 3 AnonEnv; 4 get_anon). All 1009 Rust unit tests pass; all 813 Lean tests pass; clippy clean. * Simplify ix check + add metadata-free anon mode - `ix check` becomes .ixe-only with a positional `<path>`. Drops the `--lean` compile-and-check flow and the `--env` flag; direct Lean → kernel checks remain available through `rsCheckConstsFFI` for tests. Removes the redundant `CheckIxonCmd.lean`. - New `--anon` flag runs a metadata-free kernel check: loads the .ixe via `Env::get_anon` (discards named/names/comms), enumerates work items from `env.consts` alone, and rebuilds member + ctor projection addresses deterministically via `Constant::commit`. Rejects `--consts`/`--ns`/`--consts-file` since it always checks everything. - Compiler: unwrap singleton non-inductive Muts blocks to standalone Defn/Recr. `Expr::Rec(0, univs)` already resolves correctly against a one-member block, so the wrapper was pure overhead; this keeps the env structurally uniform and matches `compile_single_def`. The anon ingress for standalones passes `mut_ctx_override = [self_id]` so self-recursive standalones still typecheck. - Kernel: anon parallel runner mirrors `run_checks_parallel_on_large _stacks` (32 workers, slow-detection, RSS tracking). Genericized `check_one_const<M>` / `check_consts_loop<M>` / `format_tc_error<M>` to share the runner. `KernelMode::meta_field_with` / `meta_field_try` gate metadata lookups in expression ingress at compile time. - Anon lazy ingress (`LazyAnonIngress` in tc.rs, helpers in ingress.rs) dedupes blocks via `kenv.blocks.contains_key(&KId<Anon>(B, ()))` before re-running `ingress_anon_block` (prevents N² re-ingestion when sibling projections fault separately within one worker check). - `Env::get_anon` now harvests `ReducibilityHints` from the otherwise-discarded Named section into a sidecar `Env::anon_hints: FxHashMap<Address, ReducibilityHints>`. Anon ingress threads these through a new `hints_override` parameter on `ingress_defn` so the lazy-delta tiebreak in `def_eq::def_rank_id` sees realistic heights instead of `Regular(0)`. Hints are performance advice, not correctness data — supplying them in anon mode preserves the metadata-free trust model. - Regenerate the canonical addresses in `PrimAddrs::new()`; singleton- unwrap changed the content hash of every self-recursive primitive (Nat.add, String, Nat.rec, …). End-to-end on compileinitstd.ixe (105,487 constants): meta: 105487/105487 in 20.6s, peak RSS 7.4GB anon: 89010/89010 in 17.8s, peak RSS 4.0GB * Clean up cargo clippy --all-targets - 11× `cloned_ref_to_slice_refs`: `&[x.clone()]` → `std::slice::from_ref(&x)` in `assumption_tree.rs` + `merkle.rs` test modules. Same allocator footprint, no clone, matches the lint's recommended idiom. - 2× `useless_vec`: `vec![0xE3, 0x00, 0x00]` → `[0xE3, 0x00, 0x00]` in serde-reject tests where the buffer is only borrowed. All 1011 unit tests pass. `cargo clippy --all-targets` is now clean. * ix check: print full content hashes + add --workers flag - The anon-mode progress / failure-log labels and the meta-mode hash-display fallback were truncating addresses to 16 hex chars (`@1f4b195aefa10e26`). Drop the `[..16]` slice so the full 64-char Blake3 hex is printed (`@1f4b195aefa10e2690d13c5b98b3d9124d3fbb5c…`), matching what tooling like `--fail-out` records and what the metadata-free workflow actually needs to identify a constant. - `--workers N` flag on `ix check` plumbs through the existing `IX_KERNEL_CHECK_WORKERS` env var that `resolve_kernel_check_workers` in `src/ffi/kernel.rs` reads. Useful for isolating per-worker memory cost (`--workers 1` lets you see the env's static footprint without per-worker overhead) and for capping concurrency in resource-tight contexts. * Mmap .ixe + cache-free LazyConstant for anon mode `lake exe ix check compilemathlib.ixe --anon` on a 3.2 GB env now peaks at ~11 GB RSS, down from ~40 GB; build_anon_work is 50× faster. Three intertwined changes: 1. Memory-map the .ixe (avoids ~3.2 GB heap copy of bytes) - Cargo: add memmap2 = "0.9". - `BytesSource` enum in `src/ix/ixon/lazy.rs` distinguishes heap-resident `Arc<[u8]>` from `(Arc<Mmap>, offset, len)` windows into a memory-mapped file. `LazyConstant::raw_bytes`, `verify_address`, `PartialEq` route through `BytesSource::as_slice` so every existing consumer is mmap-transparent. - `Env::get_anon_mmap(path)` in `src/ix/ixon/serialize.rs` opens the file, mmaps it, and stores Section-2 consts as mmap-backed `LazyConstant`s. Sections 3-4 (names + named) are still parsed transiently to harvest hints, then dropped before return. - `rs_kernel_check_anon` (`src/ffi/kernel.rs`) switches to `get_anon_mmap`; the old `std::fs::read` + `Env::get_anon` heap path is gone for the anon FFI. 2. Strip the persistent `LazyConstant` parsed-Constant cache - `LazyConstant.cache` was `Arc<OnceLock<Arc<Constant>>>` and accumulated forever — for mathlib that meant ~30 GB of parsed `Arc<Expr>` trees pinned in the env across the entire run. - New shape: `cache: Option<Arc<Constant>>`. Populated only by `from_constant` (compile-side, where we already own the parsed value). `from_bytes`/`from_mmap_slice` set `None`, and their `get()` parses fresh on every call without storing. - Re-parse cost is bounded by the existing per-worker dedup: `kenv.consts` already addresses each ingressed constant once per work item, and `clear_releasing_memory()` drops the kenv between items. The kenv is now the only persistent materialization layer. 3. `LazyConstant::peek_variant` for `build_anon_work` - One-byte read of the outer Tag4 head to identify the `ConstantInfo` variant — no body parse, no allocation. - `build_anon_work` now dispatches on `peek_variant()`; only `Muts` blocks trigger `lc.get()` (we need the member list for projection-address enumeration), and that `Arc<Constant>` drops at the end of the match arm. - Previously every constant was fully materialized at startup just to read its variant tag. With ~95% of the env being standalone/projection, the work-enumeration pass now skims the env at near-IO speed. Test updates: - `mmap_slice_roundtrips` (already added with the earlier mmap scaffolding) exercises the mmap window roundtrip. - New `from_bytes_does_not_cache` and `from_constant_clones_share_cache` document the new caching contract. - `peek_variant_*` tests cover every variant + empty-bytes and unknown-flag error paths. - `lazy_sparsity_only_materializes_closure` in `src/ix/ixon/env.rs` reframed to assert BFS-of-closure correctness instead of cache side-effects (`is_materialized()` no longer fires for lazy loads). End-to-end on compilemathlib.ixe (--anon, 32 workers): before: 640658/640658 in 266s, peak RSS ~40 GB after: 640658/640658 in 223s, peak RSS 11.3 GB * Address review findings: correctness + small refactors Code-review pass over the anon-mode / mmap / cache-strip work. This commit lands the highest-value subset — the correctness fixes and the small refactors that prevent future drift — and leaves the larger items (format-version bump, sealed marker trait, AnonEnv audit) for separate follow-ups. Correctness: - Verify per-constant address on load (#1). `LazyConstant::verify_address` existed but was never invoked from `Env::get`, `get_anon`, or `get_anon_mmap`. The env-level merkle root catches missing/extra entries but not byte-tampering of a constant whose key is intact; without this check, corruption surfaced much later inside `LazyConstant::get` with a misleading parse error. Inline the `Address::hash(bytes) == addr` check in each loader's Section-2 loop. Added 3 corruption-detection tests (`env_const_bytes_tampering_*`). This required updating a handful of existing tests that stored constants under fake `Address::hash(b"a")` keys instead of their true content hashes — round-tripping such envs now correctly rejects. Added a `store_canonical(env, c) -> Address` test helper for the canonical pattern, and `*_discriminator(refs, n)` so tests can produce content-distinct constants when the same ref-set would otherwise collide. - Hard-error in `ixon_env_to_decoded` on parse failure (#6). `src/ffi/ixon/env.rs` used `filter_map` to silently drop any const whose bytes failed `LazyConstant::get` — the Lean caller had no signal of the lost entries. Switch to a Result-collecting loop and propagate the first parse error; update both FFI call sites (`rs_de_env`, `rs_de_env_anon`). - Doc fixes (#4, #5). `docs/Ixon.md`'s AssumptionTree section described only `Leaf=0x00` and `Node=0x01`, missing `Padding=0x01` (and the docs' `Node` tag collided with the real `Padding` tag — the real `Node` is `0x02`). Also: the env-section table cell said "opt-tagged merkle root" but the implementation writes a bare 32-byte address. Refactors that prevent future drift: - Canonical projection-address helpers (#9). Four production sites reconstructed `Constant::new(IxonCI::{D,I,R,C}Prj{...}).commit().0` by hand; the anon pipeline silently breaks if any one drifts. Extract `defn_proj_address` / `indc_proj_address` / `recr_proj_address` / `ctor_proj_address` (plus `_constant` variants) in `src/ix/ixon/constant.rs`. Update `compile.rs`, `compile/mutual.rs`, and the anon `*_proj_addr` helpers to call them. - `verify_proj_addr_in_env` helper (#21). `ingress_anon_block` had the same "computed-address not in env" check repeated four times (DPrj/RPrj/IPrj/CPrj). DRY into one helper that produces a consistent error format. UX: - Anon display labels switched to `#<hex>` everywhere (#18). Rust was emitting `@<hex>` in progress/fail-out; Lean's `runCheckAnon` was emitting `#{i}` (result index). Standardize on `#<hex>` so the CLI failure summary is joinable with the fail-out file. This required exposing addresses per result slot to Lean — `rsCheckAnonFFI` now returns `Array (String × Option CheckError)` pairing each result with its content-address hex string. Rust builds the pairs via a new `build_anon_result_array` helper. - Stale "check-ixon" header in FailureLog (#16). One-line rename in the doc comment + `# ix check-ixon failures` writeln to `# ix check failures`. Tests: - `testPrimitivesParity` (PrimAddrs regen-parity test). Catches silent drift between hardcoded primitive addresses in `PrimAddrs::new()` and what `rsCompileEnvFFI` produces from the live Lean primitives. Plumbing: new `PrimAddrs::lean_parity_table()` returns `(lean_name, hex)` pairs; new `rs_prim_addrs_canonical` FFI exposes them to Lean; new test in `BuildPrimitives.lean` iterates `kernelPrimitives`, compares against the hardcoded table, and fails with a printable diff on mismatch (with instructions to regenerate). Skipped: `eagerReduce` — synthetic kernel marker whose PrimAddrs value (`0xff…3`) intentionally diverges from its compiled content hash (which collides with `id`). - 3 new corruption-detection tests in `serialize.rs` exercise the Section-2 verify check for `Env::get`, `Env::get_anon`, `Env::get_anon_mmap`. Verification: `cargo test --lib` 1025 passing (3 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe` 105487/105487 in ~21s, `--anon` 89010/89010 with peak RSS 1.4 GB, `.lake/build/bin/IxTests --ignored rust-kernel-build-primitives` shows both `build primitives dump` and `primitive address parity (PrimAddrs vs live compile)` pass. Out of scope (deferred to follow-ups): - #2 Format version bump (Env::FLAG) - #3 rs_kernel_check_consts_anon still uses Env::get on main - #8 Sealed marker trait for the lazy_anon transmute - #11/#12 AnonEnv audit (vestigial wrapper) - #13, #15, #17, #19, #20, #22-25 Various quality cleanups * Round-2 review fixes: mmap defense + small cleanup Six small items from the round-2 review of the anon-mode work: - N1: stat-at-open defense in `Env::get_anon_mmap` (`src/ix/ixon/serialize.rs`). Capture the file size before mmap; if the kernel's mapped length disagrees (file truncated between open and map) bail with a clear error instead of letting workers SIGBUS deep in `LazyConstant::get`. Truncate-in-place under a live mapping is still undefined per POSIX — documented as a caller contract — but the open-time check catches the common case. - New `get_anon_mmap_survives_file_unlink` test: load via mmap, unlink the path, then materialize both already-touched and not-yet-touched constants. Locks in the inode-retention invariant that the SIGBUS analysis depends on; a future refactor that switched to `mmap_anonymous` or copied bytes into a tmpfile would fail loudly here instead of letting workers SIGBUS in production. - #11: delete `AnonEnv::as_ixon_env_unchecked` (`src/ix/kernel/anon_env.rs`). Dead `pub(crate)` escape hatch marked `#[allow(dead_code)]` — confirmed zero callers and removed. - #13: `debug_assert!` on `OnceLock::set` for both result vectors (`src/ffi/kernel.rs`, meta + anon paths). The previous `let _ = results[idx].set(...)` silently dropped a re-set. If a future `build_*_work` dedup refactor breaks the one-write-per-slot invariant, debug builds now panic with the slot index instead of silently losing results. - #19: `allNames.contains` O(n²) preflight → `Std.HashSet` lookup (`Ix/Cli/CheckCmd.lean`). At mathlib scale (~700k env names × thousands of seed names) the previous linear scan-per-name spent measurable seconds on missing-name preflight alone. - N4: doc imprecision in `src/ix/ixon/lazy.rs` — the cache-policy preamble said the worker `KEnv` is "cleared between work items" but the actual cadence is `clear_every` items (`IX_KERNEL_CHECK_CLEAR_EVERY`, default 1 but tunable). Refined wording so the doc matches the code. Verification: `cargo test --lib` 1026 passing (1 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe --anon` 89010/89010 in 15.2s peak RSS 1.4 GB (regression-clean). Out of scope, separate follow-ups: - #2 Format version bump - #3 rs_kernel_check_consts_anon zkPCC FFI (Env::get → mmap) - #8 Sealed marker trait for the lazy_anon transmute - #10 child_arena closure side-effect (widen MField to tuple) - #12 AnonEnv duplicates bfs_refs / transitive_deps_excl - #14 anon worker bypasses M-generic display helpers - #17 Anon --fail-out docstring claims --consts-file compat - #20 Singleton-unwrap duplicated across compile paths - #22-24 Lean `partial def`, `.get!` in tests, unguarded as-u64 casts - N2/N3 Lean is_materialized() callers + compute_const_size_breakdown - Test gaps: rs_kernel_check_anon integration, concurrent mmap, verify_address-on-mmap-corruption, etc. * rustfmt * Tests: derive RawConst addrs from content (verify_address fix) The `Address::hash(bytes) == addr` defense added in f792102 rejects `RawConst { addr, const }` pairs where `addr` is uncorrelated with `serConstant const`. The Rust-side tests in `serialize.rs` were updated at the time (`store_canonical` helper), but three Lean-side sites still produced mismatched pairs and surface now as: × ∃: Env serialization Lean==Rust × ∃: serde RawEnv with data × ∃: serde RawEnv roundtrip deserialization failed: rs_de_env: Env::get: const at idx 0 bytes hash to 6110b739… but stored under b177ec1b… Three fixes, all derive `addr := Address.blake3 (serConstant c)`: - `Tests/Gen/Ixon.lean::genRawConst` — property-test generator was `RawConst.mk <$> genAddress <*> genConstant` (uncorrelated). Now generates the const first, then derives addr from `serConstant`. - `Tests/FFI/Lifecycle.lean::serdeTests` (`withData` unit case) — was using one `testAddr := Address.blake3 #[1,2,3]` for both the const and unrelated blob/comm slots. Split into `testAddr` (still used for the content-hash-free blob/comm) and `testConstAddr := Address.blake3 (serConstant testConst)`. Added a name entry for the new canonical addr. - `Tests/FFI/Lifecycle.lean::genSerdeRawEnv` — the "pool of addresses" pattern picked const addrs from a random pool that couldn't possibly match content hashes. Restructured: consts derive canonical addrs from content, each gets its own name-table entry appended to the pool-derived entries so the serde pipeline's "all addresses resolvable" invariant still holds. * Regenerate Aiur primitive addresses for singleton-unwrap Compiler's singleton non-inductive Muts unwrap (12630aa) changed the content hashes of 48 primitives (`Nat.rec`, `Nat.add`, `Nat.mul`, …). The Aiur kernel's `Ix/IxVM/Kernel/Primitive.lean` still held the pre-unwrap blake3 bytes, so primitive dispatch missed every affected constant and fell through to structural `Nat.rec` unfolding. Tests like `IxVMPrim.nat_mul_big` then drove millions of Nat.succ unfoldings inside the Aiur trace, OOMing the prover. Addresses regenerated from `PrimAddrs::lean_parity_table()` (which the branch had already updated in `src/ix/kernel/primitive.rs`). * Fix standalone Recr ingress: typ-based inductive lookup `find_matching_block_addr` heuristic in the standalone-Recr branch of `build_convert_inputs_walk` picks the inductive block by matching ctor count to rule count. When multiple in-scope inductives share the same ctor count, it returns the wrong block — the stored rules' `ctor_idx` then points to a sibling's ctor, while `populate_rules` (canonical) derives `ctor_idx` from the right inductive's `ctor_indices`. The mismatch surfaces as `compare_rules`'s `assert_eq!(s_ctor, c_ctor)` panic (e.g. `38 != 40` for `IxVMPrim.nat_land_lit`). Switch the standalone-Recr path to the same typ-based resolution `build_aux_recr_ctor_idxs` already uses for aux Recr blocks: peel `params + motives + minors + indices` foralls of `recr.typ`, take the major's head Ref, look it up in `refs` to get the inductive's address, then resolve `IPrj.block` for the Muts wrapper. Slice ctor positions for the specific member via `extract_member_ctor_idxs`. Also store the resolved Muts `block_addr` in `CKRecr` (was passing the heuristic's wrong address through to `convert_recursor`). --------- Co-authored-by: Arthur Paulino <arthurleonardo.ap@gmail.com> Co-authored-by: Samuel Burnham <45365069+samuelburnham@users.noreply.github.com>

plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed).

* Aiur Bytecode → Rust codegen for the IxVM kernel Adds a new Aiur pipeline stage that translates `Bytecode.Toplevel` into a Rust source module, one `fn aiur_fn_N` per Aiur function. The generated code mirrors `src/aiur/execute.rs`'s QueryRecord side effects exactly: same `function_queries.insert` timing, same cache-hit multiplicity bumps, same memory-queries insertion order, same `bytes{1,2}_queries` updates, same IO sequencing. Per-witness trace hashes match the interpreter byte-for-byte on every const tested (`Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). * `Ix/Aiur/Stages/Codegen.lean`: structured Rust IR (`RustExpr` / `RustStmt` / `RustItem` / `MatchArm` / labeled blocks) plus a formatter. Codegen is a structural walk over the Bytecode AST, no string templating. `EmitM = StateM EmitState` threads `(nextVal, nextLabel)` so per-ValIdx Rust locals and per-MC labels are fresh. * Aiur's `map: Vec<G>` is gone in the generated kernel: every ValIdx becomes a Rust local `__v_{i}: G`. Match-arm bodies snapshot `nextVal` on entry so per-arm allocations don't leak to siblings. `MatchContinue` / `Yield` use labeled-block + `break 'label [G; OUT_SIZE]` to bubble yielded values out and rebind them at outer scope. * Deep Aiur recursion uses `stacker::maybe_grow(64 KiB, 4 MiB, …)` at the top of every generated fn, so the native call stack grows on demand rather than pre-reserving a giant thread stack. Verified against shard 24 of the 64-way `init.ixes` partition, which SEGFAULTs without it. * `ix codegen`: new CLI command. Compiles the IxVM Aiur source, walks the bytecode, writes the generated Rust to a fixed path (`src/ix/aiur_ixvm.rs`). The output path is hard-coded; no override. * `src/ix/aiur_ixvm.rs`: the generated kernel, 743 `aiur_fn_*` + `execute_generated` dispatch. Auto-regenerated; do not edit. `#![cfg_attr(rustfmt, rustfmt::skip)]` + a broad `#![allow(unused_*, non_snake_case, clippy::all)]` since the layout is for the compiler, not humans. * `src/ix/aiur_ixvm_runner.rs`: `execute_ixvm(toplevel, fun_idx, args, io_buffer) -> Result<(QueryRecord, Vec<G>), ExecError>`. Same return shape as `Toplevel::execute`, but routes through `execute_generated`. * `src/aiur/synthesis.rs`: `AiurSystem::prove_ixvm(...)` — same shape as `prove`, but the execute step calls `execute_ixvm`. Verification-compatible (proofs from one path verify under the other). * `src/aiur/execute.rs`: helpers exposed for the codegen'd kernel (`bytes{1,2}_execute` → `pub(crate)`, `unconstrained_big_uint_div_mod_helper` extracted, `CodegenBytes{1,2}{Op,}` aliases re-exported, `QueryRecord::new` → `pub(crate)`, `ExecError::InvalidFunIdx` added). * `src/ffi/aiur/protocol.rs`: two new FFI exports — `rs_aiur_toplevel_execute_ixvm` and `rs_aiur_system_prove_ixvm`. Same wire format as the existing `_execute` / `prove` exports. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.executeIxVM` — mirror of `execute`. * `Ix/Aiur/Protocol.lean`: `AiurSystem.proveIxVM`. * `Ix/Cli/CheckCmd.lean`: `runCompiled` now dispatches through `executeIxVM`. The Rust bytecode interpreter is no longer reachable from `ix check` (the Lean-side `--interp` fallback stays for richer error diagnostics). * `Ix/Cli/ProveCmd.lean`: `proveOne` uses `proveIxVM`. * `Cargo.toml`: `stacker = "0.1"`. For witnesses where execute is the dominant work (single-const checks via `ix check 'X'`), the codegen'd kernel is ~1.6× faster than the bytecode interpreter end-to-end (measured on `Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). Tiny consts are within ~10% (stacker probe overhead amortises). Peak RSS is within a few MB of the interpreter — stacker grows the stack on demand, no giant upfront reservation. For shard-driven `ix check --shard K` runs the speedup is invisible: per-shard wall time is dominated (~92%) by `buildShardCheckEnvWitness` in Lean, NOT by the kernel execute (~8%). Cutting witness construction is the next lever. `Nat.add_comm` proof produced via `proveIxVM` verifies under the existing `AiurSystem::verify`. End-to-end: lake exe ix prove 'Nat.add_comm' → proof addr 0d0ab0f9… lake exe ix verify 0d0ab0f9… → ok * Move shard witness construction to Rust + parallelise Replaces `IxVM.ClaimHarness.buildShardCheckEnvWitness` (Lean side, ~92% of shard wall time on heavy partitions) with a Rust port that builds the `aiur::execute::IOBuffer` directly, without per-byte boxing into Lean `Aiur.G` values. The two hot phases run on rayon: * Closure walk: each owned addr's transitive `Constant.refs` + projection-block traversal runs on its own thread; results are deduped through a `dashmap::DashSet`. * Byte-to-G conversion: per-const `(key, data)` tuples are built in parallel chunks of 256; final IOBuffer assembly (channel arena append + key→`IOKeyInfo` map insert) runs serially because the arena index is monotonic. * `src/ix/aiur_ixvm_witness.rs` (new): `build_shard_check_env_witness` produces `(claim, claim_digest_input, io_buffer)` ready to feed to `execute_ixvm`. Mirrors the 6-channel layout documented in `Ix/IxVM/ClaimHarness.lean` (claim / asm tree / const bytes / Defn hint / blob discriminator / blob raw bytes). * `src/ffi/aiur/protocol.rs`: new FFI `rs_aiur_toplevel_shard_check_ixvm` bundles witness build + `execute_ixvm` into one cross-language trip. Returns the same `(output, ioBuffer, queryCounts)` shape as `rs_aiur_toplevel_execute_ixvm`, so the Lean shim stays drop-in compatible. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.shardCheckIxVM` wraps the FFI. * `Ix/Cli/CheckCmd.lean`: shard-mode `ix check` (`--ixe + --ixes`) now dispatches through `runShardOwnedNative` → `shardCheckIxVM`, bypassing the Lean witness builder. Single- shard and whole-partition paths share the fast route; the legacy `runShardCheckManifest` / `runShardCheckAll` callbacks stay live for `--interp` only. Shard 26 of the 64-way `init.ixes` partition, on the same host: | Variant | total | speedup | | --- | --: | --: | | Lean witness + bytecode-interp (baseline)| 1029 s | 1.0× | | Lean witness + codegen kernel | 1015 s | 1.01× | | Rust witness (serial) + codegen | 101 s | 9.95× | | Rust witness (parallel) + codegen | 80 s | 12.9× | The serial Rust witness collapses the 935 s of Lean per-byte boxing into ~23 s; rayon hides that 23 s under the codegen-kernel execute (~78 s) so witness build effectively becomes free at the shard level. FFT cost matches the bytecode interpreter exactly (107_006_963_281), so the QueryRecord layout is bit-identical across all four variants. * Closure walk uses single-source BFS with the global visited set shared via `DashSet::contains` checks before pushing to the per-thread stack — avoids redundant work without a per-owned union step. * Per-channel ordering: each chunk's partial `ChannelEntries` feeds the serial fold in iteration order, so channel arena contents match the Lean side's exactly. Test still relies on trace-hash parity from the earlier codegen commit, which exercised the same path through the bytecode interpreter and the codegen kernel. * Coverage check is currently skipped on the IxVM-native whole-partition path (`runShardManifestAllNative`); legacy `runShardCheckAll` still does it for `--interp`. Re-enable separately if needed. * Codegen: value-returning byte ops, dead-fold the unconstrained plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed). * Wire Rust witness fast path through every check/prove entry point Per-claim mode (a `Claim.check addr none`) builds a closure rooted at the target address. That closure can be the whole environment when the target is a heavily-shared root constant. The Lean witness builder paid per-byte boxing into `Aiur.G` for every byte in that closure — the same cost we already eliminated for shard mode. # Surface Five new FFIs, all bundling witness build + execute_ixvm (and STARK prove where applicable) into one cross-language trip so the `IOBuffer` never crosses the boundary mid-pipeline: * `rs_aiur_toplevel_check_addr_ixvm` — `Bytecode.Toplevel.checkAddrIxVM (toplevel) (funIdx) (ixePath) (addrBytes)`: per-claim check; builds witness for `Claim.check addr none` via `build_claim_check_witness`, runs `execute_ixvm`. Same return shape as `shardCheckIxVM`. * `rs_aiur_toplevel_check_env_bytes_ixvm` — same as above but takes the env as a serialized byte blob (`Ixon.serEnv`) instead of a `.ixe` path. Used by the compiled-Lean-env code path (`ix check NAME` without `--ixe`), where the env is built in Lean memory. * `rs_aiur_system_prove_addr_ixvm` — `AiurSystem.proveAddrIxVM (...)`: per-claim prove (witness + execute + STARK prove all in Rust). * `rs_aiur_system_prove_env_bytes_ixvm` — bytes-blob counterpart. * `rs_aiur_system_shard_prove_ixvm` — `AiurSystem.shardProveIxVM (...)`: per-shard prove end-to-end in Rust. `build_claim_check_witness` lives next to `build_shard_check_env_witness` in `src/ix/aiur_ixvm_witness.rs` and uses the same parallel closure walk + parallel byte→G conversion. The bytes-blob FFIs additionally harvest `anon_hints` from each `Def` named entry after `Env::get` — `Env::get` (full form, used by the bytes-blob path) doesn't populate `env.anon_hints` the way `Env::get_anon` does, but the kernel's `verify_claim` reads ch 3 (Defn reducibility hints) so the hints must end up in the env. This mirrors the harvest pass already inside `get_anon` at `src/ix/ixon/serialize.rs:1683`. # Wiring `Ix.Cli.CheckCmd.WitnessSource`: a three-arm discriminated union threaded through `forEachClaim` (the shared check/prove driver): * `.native ixePath addr` — `.ixe`-backed env, Rust mmap. * `.nativeBytes envBytes addr` — Lean-memory env serialized to bytes, Rust decodes via `Env::get`. * `.lean witness` — pre-built `ClaimWitness` (fallback). `runCompiled` and `proveOne` dispatch on the source. # Coverage | Command | Path | |---------------------------------------|-----------------------------------------------| | `ix check NAME --ixe` | **`checkAddrIxVM`** (new) | | `ix check NAME` (no `--ixe`) | **`checkEnvBytesIxVM`** (new) | | `ix check --claim hex --ixe` | `checkAddrIxVM` for `check addr none` | | `ix check --ixes --shard K` | `shardCheckIxVM` (existed) | | `ix check --ixes` | `shardCheckIxVM` (existed) | | `ix prove NAME --ixe` | **`proveAddrIxVM`** (new) | | `ix prove NAME` (no `--ixe`) | **`proveEnvBytesIxVM`** (new) | | `ix prove --ixes --shard K` | **`shardProveIxVM`** (new) | | `ix prove --ixes` | **`shardProveIxVM`** loop (new) | | `Benchmarks/Typecheck.lean` Phase 1 | `checkAddrIxVM` / `executeIxVM` | | `Benchmarks/Typecheck.lean` Phase 2 | `proveAddrIxVM` / `proveIxVM` | | `Benchmarks/IxVM.lean` | `proveIxVM` (was `prove`) | `--interp` route is preserved: it materialises any `WitnessSource` back into a `ClaimWitness` before driving the Aiur source interpreter. `--claim <hex>` over non-`check addr none` variants (`eval` / `reveal` / `contains` / `checkEnv-with-asm`) still uses the Lean witness builder. # Sanity * `ix check Nat.add_comm --ixe init.ixe` (warm): 3.1 s wall, FFT = 23_603_449. * `ix check Nat.add_comm` (compiled env via bytes blob, warm): 2.4 s wall, FFT = 23_603_449. # Known overheads (tracked for a follow-up redesign) * **Per-claim env re-parse on `--ixe + many names`**: each FFI call re-runs `Env::get_anon_mmap` on the same path. Mathlib-scale iteration pays the lazy-index build N times instead of once. * **`--interp + .nativeBytes` round-trip**: Lean serializes the compiled env then `materialise` deserializes it back, just to feed `runInterp`. * **`runShardProveNative` claim reconstruct**: redoes the closure walk + canonical `AssumptionTree` build Lean-side after `shardProveIxVM` already computed the same claim internally. A future `EnvHandle`-based API would close all three: build the env once into a Rust-owned handle, pass the handle to every per-claim/per-shard FFI, and have prove FFIs return the serialized claim bytes. * EnvHandle: Rust-owned env shared across per-claim / per-shard FFI calls Before: every per-claim and per-shard FFI re-parsed the Ixon env (`Env::get_anon_mmap` per call). On `--ixe + many names` / all-shards prove, this is the dominant overhead for iteration-heavy workflows — O(num_consts) lazy-index build × N targets. After: the env lives once per CLI invocation in a Rust-owned `EnvHandle`. Lean holds an opaque `Aiur.EnvHandle` reference and threads `@& EnvHandle` through every per-target FFI call. The env is parsed exactly once at handle construction; downstream calls share it. # Surface Five new FFIs collapse the previous six per-call variants (`checkAddrIxVM`, `checkEnvBytesIxVM`, `shardCheckIxVM`, `proveAddrIxVM`, `proveEnvBytesIxVM`, `shardProveIxVM` — all deleted): * `rs_aiur_env_handle_from_ixe(path) → LeanExternal<EnvHandle>`: mmap-load via `Env::get_anon_mmap`. Anon parser already harvests `anon_hints`; no post-pass. * `rs_aiur_env_handle_from_bytes(blob) → LeanExternal<EnvHandle>`: decode `Ixon.serEnv`-shape blob via `Env::get` + harvest `anon_hints` from each `Def` named entry. Used by the compiled-Lean-env path. * `rs_aiur_toplevel_check_addr_with_env(toplevel, fun_idx, handle, addr_bytes)`: per-claim check. Reuses handle's parsed env. * `rs_aiur_toplevel_shard_check_with_env(toplevel, fun_idx, handle, owned_blob)`: per-shard check. * `rs_aiur_system_prove_addr_with_env(system, fri, fun_idx, handle, addr_bytes) → (claim_bytes, proof, ioBuffer)`: per-claim prove. Rust serializes the reconstructed `Ix.Claim` via `ixon::Claim::put` so Lean can deserialize via `Ixon.runGet Ix.Claim.get` directly — no closure walk + canonical `AssumptionTree` recomputation Lean-side. * `rs_aiur_system_shard_prove_with_env(system, fri, fun_idx, handle, owned_blob) → (claim_bytes, proof, ioBuffer)`: per-shard prove. # Lean * `Aiur.EnvHandle` opaque type with `fromIxe` / `fromBytes`. * `Aiur.Bytecode.Toplevel.checkAddrWithEnv` / `shardCheckWithEnv`, `Aiur.AiurSystem.proveAddrWithEnv` / `shardProveWithEnv`. * `Ix.Cli.CheckCmd.Target` replaces `WitnessSource`: ```lean inductive Target where | addr (a : Address) -- Claim.check addr none | shard (owned : Array Address) -- Claim.checkEnv | leanW (w : ClaimWitness) -- --interp, --claim non-check ``` `runCompiled` / `proveOne` take `(envHandle?, target)`. The envHandle is `none` only for `.leanW` (`--interp` legacy path). * `runShardOwnedNative` and the manifest-driver helpers take the envHandle as a parameter. Single-shard mode builds it once; all-shards mode reuses the same handle across every shard's FFI call (eliminates per-shard re-mmap). * `runShardProveNative` deserializes the wire claim bytes via `Ixon.runGet Ix.Claim.get` instead of re-running `shardCheckEnvClaim`. # Benchmarks `Benchmarks/Typecheck.lean` builds the envHandle once before Phase 1, reuses it across Phase 1 (execute) + Phase 2 (prove). Both phases now go through `checkAddrWithEnv` / `proveAddrWithEnv` on the full-closure path. `--subject-only` still uses Lean `buildVerifyConst` + `executeIxVM` / `proveIxVM` (witnesses intentionally small there). # New crate `crates/ix/src/env_handle.rs` — the `EnvHandle` struct + constructors live in the existing `ix` crate next to the witness builder and codegen runner. # Sanity * Multi-target check (warm): `ix check --ixe init.ixe Nat.add_comm Nat.add` → 3.3 s wall, single envHandle shared across both targets. * Shard 26 check (warm): `ix check --ixe init.ixe --ixes init.ixes --shard 26` → ~78 s wall, FFT = 107_006_963_281 (parity preserved). # Coverage | Command | Path | |--------------------------------------|-----------------------------------| | `ix check NAME --ixe` | `checkAddrWithEnv` | | `ix check NAME` (no `--ixe`) | `checkAddrWithEnv` + fromBytes | | `ix check --claim hex --ixe` | `checkAddrWithEnv` for check-none | | `ix check --ixes --shard K` | `shardCheckWithEnv` | | `ix check --ixes` | `shardCheckWithEnv` × all shards | | `ix prove NAME --ixe` | `proveAddrWithEnv` | | `ix prove --ixes --shard K` | `shardProveWithEnv` | | `ix prove --ixes` | `shardProveWithEnv` × all shards | | `Benchmarks/Typecheck` Phase 1+2 | `checkAddrWithEnv` / `proveAddrWithEnv` | Only `--interp` and `--claim hex` over a non-`check addr none` persisted claim still build a Lean `ClaimWitness`; both go via the `.leanW` target arm. * ix codegen: add --check flag for CI; gate stale generated kernel `ix codegen --check` compares the emitted Rust source against the on-disk `crates/ix/src/aiur_ixvm.rs` and exits 0 if identical, 1 otherwise. No write side effect. ~2 s warm; fast enough to gate on every PR. Wired into the `lean-test` CI job so a forgotten regen on a kernel-touching PR fails CI instead of merging stale generated code that drifts from the Bytecode → Rust emitter. * ix check: --interp {source|bytecode} to bypass codegen kernel Two interpreter modes behind a single `--interp` flag: * `--interp source`: Aiur source interpreter (`Aiur.runFunction` over the source-level `Decls`). Richer per-step error diagnostics. * `--interp bytecode`: generic Aiur bytecode interpreter (`Bytecode.Toplevel.execute`, `rs_aiur_toplevel_execute` route). Skips the `ix codegen` + `cargo build --release` cycle needed after editing `Ix/IxVM/*.lean` — the bytecode is rebuilt Lean-side at exe load. Slower per-check than the codegen kernel; ideal for tight iteration on the IxVM source. * omit the flag entirely for the native codegen kernel (default). Invalid values (e.g. `--interp foo`) fail fast with a clear error. # Plumbing The `check_addr_with_env` and `shard_check_with_env` FFIs take a `use_bytecode: bool` and dispatch via a shared `dispatch_execute` helper. `--interp bytecode` sets it to `true`. The `.leanW` arm of `runCompiled` also picks between `Bytecode.Toplevel.execute` and `executeIxVM` based on the same flag. The `--interp source` path requires a Lean `ClaimWitness`. The existing driver had switched to `.addr`/`.shard` targets against the Rust-owned `EnvHandle`; source-interp had no way to materialise those. `forEachClaim` now takes a `forceLeanWitness : Bool` — when true, it builds a Lean witness for every target (via `mkWitness` / `loadIxonEnv` for the compiled-Lean-env path) and passes `.leanW` so `runInterp` can consume it directly. # Sanity * `ix check Std.Time.Week.Offset.ofMilliseconds` (warm): 9.3 s wall, FFT = 12_430_516_949 (codegen kernel). * `ix check --interp bytecode Std.Time.Week.Offset.ofMilliseconds` (warm): 14.6 s wall, FFT = 12_430_516_949 (bytecode interp). * `ix check --interp source Eq` (warm): 6.3 s wall, output `Eq: ()`. * `ix check --interp garbage Nat.add_comm`: exits 1 with clear error. * IxVM codegen: regen + rebase-fixups after KLevel port Aligns the ap/codegen-ixvm-native stack with main's KLevel pointer port (b84a500) and re-lands clippy-clean under the workspace lint set: - Regen crates/ix/src/aiur_ixvm.rs from Aiur source (KLevel = &KLevelNode changes the generated fn shapes; ~26k lines net churn). - Extend the generated file's #![allow(...)] header (via Ix/Aiur/Stages/Codegen.lean) with clippy::ptr_as_ptr, clippy::match_same_arms, and clippy::large_types_passed_by_value — three lints the codegen's straight-line style trips deterministically on every regen and that clippy::all doesn't cover. - rustfmt reflow in crates/aiur/src/execute.rs and a handful of ix / ffi files that CI now enforces via the ap/aiur-aux-recursor-parity toolchain. - Fix 4 real clippy warnings uncovered under the workspace lint set: * env_handle.rs: clone-on-Copy → deref (ReducibilityHints is Copy). * aiur_ixvm_witness.rs: collapse if-let-if, drop needless continue, slice::from_ref instead of &[x.clone()]. * aiur_ixvm_runner.rs: keep execute_ixvm's Vec<G> arg (required by AiurSystem::prove_ixvm's fn-pointer bound) + local #[allow( clippy::needless_pass_by_value)] with a comment explaining why. * ffi/aiur/protocol.rs: unqualify aiur::G/IOBuffer/QueryRecord after the workspace lint added unused_qualifications; is_multiple_of() over % 0. Verified: cargo clippy --workspace --all-targets --all-features --D warnings clean; lake test -- --ignored ixvm passes. * ix-check + ixvm tests: coverage gate, --interp source fix, codegen parity + rename ix crate to ixvm-codegen Four review fixes on the codegen-ixvm-native stack. - **Coverage gate on \`--ixes\` (all-shards, native path).** \`runShardManifestAllNative\` now calls \`shardsCover\` before running any shard, matching the pre-refactor \`runShardCheckAll\` behavior. Without this, a stale/truncated \`.ixes\` manifest makes \`ix check --ixes\` exit 0 while some env constants are never checked by any shard — the mundane trigger being: edit a definition, rebuild the \`.ixe\`, forget to re-run \`ix shard\`, and the constants that go unchecked are exactly the ones just edited. Single-shard \`--shard K\` path is unchanged (consistent with pre-refactor). \`shardsCover\` moved above \`runShardManifestAllNative\` to satisfy forward-decl ordering. - **\`ix check --ixe … --claim <hex> --interp source\` regression.** The \`claimHex\` arm in \`forEachClaim\` ignored \`forceLeanWitness\` and always mapped \`.check addr none\` claims to \`Target.addr\`. Under \`--interp source\`, \`runInterp\` then rejects \`.addr\` with "\`--interp requires a Lean witness; addr/shard targets unreachable here\`". Now mirrors the sibling name-based arms and routes through the Lean witness path when \`forceLeanWitness\` is on. Only bit the common claim shape. - **Codegen ↔ bytecode parity CI test.** New \`runParityCase\` + \`parityCases\` in \`Tests/Ix/IxVM.lean\` wire every constant already listed in \`kernelCheckEntries\` plus \`kernel_unit_tests\` through BOTH \`Toplevel.execute\` (bytecode) and \`Toplevel.executeIxVM\` (codegen'd Rust), and diff the returned \`(output, IOBuffer, QueryCounts)\` triples. Turns "generated kernel ≡ interpreter on QueryRecord" from reviewed-by-hand into checked-by-CI. 129 assertions run per suite invocation (43 constants × output + IOBuffer + QueryCount), all pass on this branch head. - **Rename \`crates/ix\` → \`crates/ixvm-codegen\`.** The old \`ix\` crate held only the codegen'd IxVM kernel + its runner + witness helpers. \`ix-kernel\` is the hand-written Rust ix typechecker; \`ix-common\` / \`ix-compile\` are Ix-level infrastructure. \`ixvm-codegen\` cleanly names what this crate holds. Path updates: workspace \`members\` and \`workspace.dependencies\`, ffi crate dep + \`use\` sites, Lean \`codegenOutPath\` in \`Ix/Cli/CodegenCmd.lean\`, doc reference in \`Ix/Aiur/Semantics/BytecodeFfi.lean\`. \`ix codegen\` now writes to \`crates/ixvm-codegen/src/aiur_ixvm.rs\`. Verified: \`cargo check --workspace\` + \`lake test -- --ignored ixvm\` both exit 0 post-rename; parity + pinned FFT tests all pass. * cargo-deny: allow ar_archive_writer's Apache-2.0 WITH LLVM-exception Post-rename, ixvm-codegen pulls stacker → psm → ar_archive_writer (a build-time transitive). ar_archive_writer is licensed under the LLVM-exception variant of Apache-2.0 (standard for LLVM tooling; no additional runtime obligation for downstream users), which isn't on the workspace allow list. Add a per-crate exception scoped to ar_archive_writer so we don't blanket-allow the license family across the tree.

arthurpaulino · 2025-02-04T13:52:08Z

No description provided.

Lays the groundwork for measuring the kernel performance plan (plans/okay-let-s-write-a-lucky-dolphin.md) per audit §10. Counters live in a new kernel::perf module; KEnv carries a PerfCounters field that is dumped from a Drop impl when IX_PERF_COUNTERS=1 is set. Unset is the production default — every increment short-circuits via a LazyLock<bool> so the cost is a single cached branch on the hot path. Wired into the cache get sites the audit identified: - whnf_cache hits/misses (whnf.rs around 201/211) - whnf_no_delta_cache hits/misses (whnf.rs around 437/446) - infer_cache and infer_only_cache hits/misses (infer.rs around 45/51) - def_eq_cache hits/misses (def_eq.rs around 137/149) - def_eq_failure set hits/inserts (def_eq.rs around 360) - per-constant peak/avg MAX_REC_FUEL consumption (recorded in TypeChecker::reset before the next constant resets it) Verified against lake test -- kernel-check-env --ignored: 192156/192156 constants pass in 251s (4% over the 242s baseline; overhead is the LazyLock branch + atomic load in the disabled path). P1: def_rank_id replaces def_weight_id for hint-priority comparison Audit Tier 1 #3 (kernel-perf-adversarial-audit-2026-04-26.md, §4.2): the prior u32 encoding mapped Abbrev to u32::MAX-1 and saturating-added Regular(h) to h+1, which collide at h ≥ u32::MAX-2. When that happens the delta-direction logic treats Abbrev and a maximally-heavy Regular as "same height" and unfolds both, instead of preferring Abbrev as Lean does (compare(d_t->get_hints(), d_s->get_hints()) at type_checker.cpp:910). Replace the u32 weight with a (class: u8, height: u32) tuple compared lexicographically: - Opaque / Theorem / unknown → (0, 0) - Regular(h) → (1, h) (height ordering preserved within class) - Abbrev → (2, 0) (strictly above every Regular) Update the two call sites (is_def_eq lazy-delta height comparison and lazy_delta_step). The map_or default for "missing head" is preserved as (u8::MAX, u32::MAX) — the branch is dead in practice (a_delta && b_delta imply both heads are present) but kept consistent with the prior u32::MAX sentinel. Two regression tests: - def_rank_abbrev_above_saturated_regular: Abbrev outranks Regular(u32::MAX) (the previous saturation collision). - def_rank_regular_orders_by_height: height monotonically orders Regular ranks within the class. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 256s (no regression vs the 242s pre-perf-counters baseline; the +14s is from the IX_PERF_COUNTERS=unset LazyLock branch added in the prior commit, not this change). P2: peel_proj_forall fast-paths syntactic Pi in projection inference Audit Tier 1 #2 (kernel-perf-adversarial-audit-2026-04-26.md, §7.2): infer_proj's two parameter-consuming loops (param peel and field peel) called self.whnf(&r)? unconditionally per iteration, on a body mutated by subst at the previous step. The whnf cache rarely hits between iterations and each call re-traverses the substituted body. Extract a peel_proj_forall(&r, err) helper that: - tries ExprData::All(..) syntactically first (no WHNF call), and - falls back to full self.whnf(e) only when the binder isn't already syntactic Pi. This mirrors Lean's inferProj at type_checker.cpp:582–610. Both projection-inference loops now call peel_proj_forall instead of unconditional whnf. Behaviorally equivalent — same WHNF semantics on miss, no semantic change otherwise. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 258s (parity with the post-pre-work, post-P1 baseline; no measurable regression and the cache-hit-rate counters will move on tactic-heavy workloads under IX_PERF_COUNTERS=1). P3a: WhnfFlags substrate (no behavior change) Lays the foundation for the Lean4Lean architectural alignment described in plans/okay-let-s-write-a-lucky-dolphin.md. Phase 3a is substrate-only — no call site is migrated to cheap mode yet, so behavior is unchanged. Adds: - WhnfFlags { cheap_rec, cheap_proj } with FULL and CHEAP consts and is_full(). CHEAP is currently equal to FULL until Phase 3c wires it. - whnf_core_with_flags (private): the existing whnf_core impl, now threading flags into recursive calls and try_iota_with_flags. - whnf_core / whnf_core_cheap (super): FULL/CHEAP wrappers. - whnf_no_delta_with_flags (private): the existing whnf_no_delta impl with the Prj branch gated on cheap_proj — falls back to full whnf on the projected value when not cheap. - whnf_no_delta (pub) / whnf_no_delta_cheap (super): wrappers. - try_iota_with_flags: gates major-premise WHNF and string-literal constructor reduction on cheap_rec. - try_proj_app_reduce_with_flags: gates projected-value WHNF on cheap_proj. Cache reads/writes (whnf_no_delta_cache, equiv-manager second-chance) are gated on flags.is_full(): cheap callers neither read nor write the cache, preserving the invariant that any cached entry is a fully-reduced normal form. Phase 3b will inline the projection branch into whnf_core to match Lean4Lean's two-layer architecture (refs/lean4lean/Lean4Lean/ TypeChecker.lean:266, 297). Phase 3c will flip CHEAP to enable cheap_proj and migrate specific def-eq sites. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 243s (matches the 242s baseline; substrate adds no measurable overhead when CHEAP == FULL). P3b: inline projection into whnf_core (Lean4Lean architectural alignment) Move the Prj branch from whnf_no_delta_with_flags into whnf_core_with_flags so our whnf_core matches Lean4Lean's whnfCore semantics exactly (refs/lean4lean/Lean4Lean/TypeChecker.lean:284-292, 337-341). Before this commit: whnf_core — beta + zeta + iota + cheap projection (recursive whnf_core on val, no delta) whnf_no_delta — whnf_core + FULL projection (full whnf on val) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta Lean4Lean's architecture has no whnf_no_delta layer. Their whnfCore includes projection, with the cheap_proj flag deciding whether the projected value uses whnfCore (cheap) or whnf (full). After this commit: whnf_core (with WhnfFlags) — beta + zeta + iota + projection (cheap_proj controls val reduction) whnf_no_delta — whnf_core(_, FULL) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta The bare-Prj branch in whnf_no_delta_with_flags is removed — whnf_core now handles it directly. The App-of-Prj branch stays in whnf_no_delta because whnf_core's loop returns once the outermost Prj is resolved; try_proj_app_reduce_with_flags gives one more attempt at the same cheap_proj policy when the outer expression is App(Prj, ...). Pure refactor, no semantic change with CHEAP == FULL. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 270s (within noise of the 243s pre-refactor baseline). Phase 3c will flip CHEAP to enable cheap_proj=true and migrate def-eq's lazy-delta sites surgically per Lean4Lean's pattern. P3c (postponed): document the HeaderParsedSnapshot regression P3a (substrate) and P3b (Lean4Lean architectural alignment) are committed. Phase 3c — flipping CHEAP to enable cheap_proj=true and migrating the def-eq lazy-delta sites — was attempted but reproduced 5 failures on chained projections in Lean.Language.Lean.HeaderParsedSnapshot.* even after P3b inlined the projection branch into whnf_core. The substrate is left in place; CHEAP stays equal to FULL until the regression's root cause is understood. Notes on the regression for the next investigator: - Failures: HeaderParsedSnapshot.{stx,result?,metaSnap,toSnapshot,ictx}, all with 'projection: type mismatch with declared struct'. - The struct `extends` a parent, so each projection is a chained Prj whose val is itself a Prj into the parent. - The error comes from infer.rs's infer_proj at the head-vs-struct_id address compare, after FULL whnf on val_ty. That whnf is FULL, but val_ty was inferred via paths that may have consulted a def-eq cache populated under cheap mode. Possible cache-poisoning suspect: def_eq_cache writes a `false` result for inputs whose lazy-delta loop bottomed out under cheap projections. The cache key uses raw a/b hashes, not cheap-reduced shapes, so a stored `false` is indistinguishable from a FULL `false` by future readers. - Lean4Lean does not have an analogous wide def_eq_cache; their failure cache is keyed only on same-spine pairs in lazyDeltaReductionStep. Future P3c iterations should either prove the cache poisoning theory incorrect or restrict def_eq_cache writes to FULL-derived results.

* Refactor Claim ADT + add merkle infrastructure for ZK aggregation Adds the format hooks needed before recursive verification lands in Aiur. Five-variant Claim ADT with explicit assumption commitments, a canonical Blake3 merkle module, a serializable AssumptionTree for recovering leaf sets, and a Contains claim for the discharge step. Claim ADT (5 variants): - Eval { input, output, assumptions: Option<Address> } - Check { const_addr, assumptions: Option<Address> } - CheckEnv { root, assumptions: Option<Address> } - Reveal { comm, info } -- unchanged, no assumptions - Contains { tree, const_addr } -- new, for inclusion proofs Tag4 reorganized to keep everything in single-byte tags: - 0xE for env, comm, AssumptionTree, and claims (slots 0-7) - 0xF for proofs (slots 0-4; 5-7 reserved) - Comm moved from variant 5 -> 1 Matches the "Variant (0-7)" constraint documented in docs/Ixon.md. Env serialization: - Every .ixe file now carries a canonical merkle root over its consts.keys() in the on-disk header (non-optional, 32 bytes; empty const sets use the zero-address sentinel). - Two envs with the same const set produce byte-identical roots regardless of insertion order. Verified on deserialize. New modules: - src/ix/ixon/merkle.rs + Ix/Merkle.lean: canonical sorted builder, free-form merkle_join composition, membership proofs, domain separation per RFC 6962. - src/ix/ixon/assumption_tree.rs + Ix/AssumptionTree.lean: serializable merkle tree with Leaf/Padding/Node variants. canonical() builds the same shape merkle_root_canonical hashes; join() is O(1) free-form composition. - src/ix/kernel/claim.rs: builders that compute transitive-dep assumptions from an env (build_check_claim, build_eval_claim, build_check_env_claim, env_merkle_root). Other: - Extracted shared BFS walker on Env::bfs_refs + Env::transitive_deps_excl; the inlined test-feature copy in lean_env.rs now calls into it. - Lean FFI export rs_env_merkle_root for cross-impl verification of the env root. - Proof bytes for all variants are uniform opaque ZK bytes; witness data (e.g., Contains merkle paths) is prover-side scratch consumed by the ZK circuit and not transmitted on the wire. - docs/Ixon.md Tag4 tables and env section updated; .ixe extension documented. Recursive verification (the ZK proof generation for Contains and the aggregation discharge transitions) is intentionally deferred to a follow-up. Tests: 993 Rust unit tests pass (was 953 pre-refactor), 813 Lean tests pass with no failures; cargo clippy clean. * Add lazy env deserialization and anon-mode kernel FFI Two related changes to reduce kernel memory + lay groundwork for metadata-isolated typechecking. **Lazy constant deserialization.** `Env::consts` now stores `LazyConstant` (`Arc<[u8]>` + `OnceLock<Arc<Constant>>`) instead of `Constant`. Constants are materialized on first access and the structured form is cached. Sparse access patterns (single-constant typecheck, transitive_deps walk, claim builders) only parse the closure they need; non-reachable constants stay as raw bytes. The `.ixe` section-2 layout gains a Tag0 length sidecar before each constant's Tag4 bytes: old: [addr:32] [Tag4 constant bytes] new: [addr:32] [Tag0 length] [Tag4 constant bytes] The length is section-level framing, not part of the constant's content hash. `Address::hash(raw_bytes) == addr` is preserved. `Constant::put`/`Constant::get` are unchanged. The Lean side (`Ix.Ixon.putEnv`/`getEnv`) reads/writes the new format. **Anon-mode kernel FFI.** New `rs_kernel_check_consts_anon(path, addrs, quiet)` exposes anonymous-mode typechecking by content address. The kernel runs as `KEnv<Anon>` / `TypeChecker<Anon>` with every `M::MField<T>` erased to `()`, so the typechecking logic structurally cannot read metadata. Useful for zkPCC verifiers that hold only addresses. Supporting pieces: - `KernelMode::HAS_META: bool` for future compile-time gating. - `AnonEnv<'a>` wrapper (`src/ix/kernel/anon_env.rs`) exposing only consts/blobs/transitive walks — no `named`/`names`/`comms`. - `Env::get_anon` reads header + blobs + consts, parse-and-drops metadata sections (3-5), returns an `Env` with empty `named`/ `names`/`comms`. Same merkle-root verification as `Env::get`. - `rs_de_env_anon` FFI + `Ix.Ixon.rsDeEnvAnon` Lean wrapper. - `Ix.KernelCheck.rsCheckConstsAnonFFI` Lean binding. Caveat: `ixon_ingress::<Anon>` still consults `Env::named` internally to enumerate work items. The resulting `KEnv<Anon>` is metadata-free so the typechecker is anon, but full ingress-level metadata isolation is a follow-up. Tests: 16 new (9 lazy + sparsity; 3 AnonEnv; 4 get_anon). All 1009 Rust unit tests pass; all 813 Lean tests pass; clippy clean. * Simplify ix check + add metadata-free anon mode - `ix check` becomes .ixe-only with a positional `<path>`. Drops the `--lean` compile-and-check flow and the `--env` flag; direct Lean → kernel checks remain available through `rsCheckConstsFFI` for tests. Removes the redundant `CheckIxonCmd.lean`. - New `--anon` flag runs a metadata-free kernel check: loads the .ixe via `Env::get_anon` (discards named/names/comms), enumerates work items from `env.consts` alone, and rebuilds member + ctor projection addresses deterministically via `Constant::commit`. Rejects `--consts`/`--ns`/`--consts-file` since it always checks everything. - Compiler: unwrap singleton non-inductive Muts blocks to standalone Defn/Recr. `Expr::Rec(0, univs)` already resolves correctly against a one-member block, so the wrapper was pure overhead; this keeps the env structurally uniform and matches `compile_single_def`. The anon ingress for standalones passes `mut_ctx_override = [self_id]` so self-recursive standalones still typecheck. - Kernel: anon parallel runner mirrors `run_checks_parallel_on_large _stacks` (32 workers, slow-detection, RSS tracking). Genericized `check_one_const<M>` / `check_consts_loop<M>` / `format_tc_error<M>` to share the runner. `KernelMode::meta_field_with` / `meta_field_try` gate metadata lookups in expression ingress at compile time. - Anon lazy ingress (`LazyAnonIngress` in tc.rs, helpers in ingress.rs) dedupes blocks via `kenv.blocks.contains_key(&KId<Anon>(B, ()))` before re-running `ingress_anon_block` (prevents N² re-ingestion when sibling projections fault separately within one worker check). - `Env::get_anon` now harvests `ReducibilityHints` from the otherwise-discarded Named section into a sidecar `Env::anon_hints: FxHashMap<Address, ReducibilityHints>`. Anon ingress threads these through a new `hints_override` parameter on `ingress_defn` so the lazy-delta tiebreak in `def_eq::def_rank_id` sees realistic heights instead of `Regular(0)`. Hints are performance advice, not correctness data — supplying them in anon mode preserves the metadata-free trust model. - Regenerate the canonical addresses in `PrimAddrs::new()`; singleton- unwrap changed the content hash of every self-recursive primitive (Nat.add, String, Nat.rec, …). End-to-end on compileinitstd.ixe (105,487 constants): meta: 105487/105487 in 20.6s, peak RSS 7.4GB anon: 89010/89010 in 17.8s, peak RSS 4.0GB * Clean up cargo clippy --all-targets - 11× `cloned_ref_to_slice_refs`: `&[x.clone()]` → `std::slice::from_ref(&x)` in `assumption_tree.rs` + `merkle.rs` test modules. Same allocator footprint, no clone, matches the lint's recommended idiom. - 2× `useless_vec`: `vec![0xE3, 0x00, 0x00]` → `[0xE3, 0x00, 0x00]` in serde-reject tests where the buffer is only borrowed. All 1011 unit tests pass. `cargo clippy --all-targets` is now clean. * ix check: print full content hashes + add --workers flag - The anon-mode progress / failure-log labels and the meta-mode hash-display fallback were truncating addresses to 16 hex chars (`@1f4b195aefa10e26`). Drop the `[..16]` slice so the full 64-char Blake3 hex is printed (`@1f4b195aefa10e2690d13c5b98b3d9124d3fbb5c…`), matching what tooling like `--fail-out` records and what the metadata-free workflow actually needs to identify a constant. - `--workers N` flag on `ix check` plumbs through the existing `IX_KERNEL_CHECK_WORKERS` env var that `resolve_kernel_check_workers` in `src/ffi/kernel.rs` reads. Useful for isolating per-worker memory cost (`--workers 1` lets you see the env's static footprint without per-worker overhead) and for capping concurrency in resource-tight contexts. * Mmap .ixe + cache-free LazyConstant for anon mode `lake exe ix check compilemathlib.ixe --anon` on a 3.2 GB env now peaks at ~11 GB RSS, down from ~40 GB; build_anon_work is 50× faster. Three intertwined changes: 1. Memory-map the .ixe (avoids ~3.2 GB heap copy of bytes) - Cargo: add memmap2 = "0.9". - `BytesSource` enum in `src/ix/ixon/lazy.rs` distinguishes heap-resident `Arc<[u8]>` from `(Arc<Mmap>, offset, len)` windows into a memory-mapped file. `LazyConstant::raw_bytes`, `verify_address`, `PartialEq` route through `BytesSource::as_slice` so every existing consumer is mmap-transparent. - `Env::get_anon_mmap(path)` in `src/ix/ixon/serialize.rs` opens the file, mmaps it, and stores Section-2 consts as mmap-backed `LazyConstant`s. Sections 3-4 (names + named) are still parsed transiently to harvest hints, then dropped before return. - `rs_kernel_check_anon` (`src/ffi/kernel.rs`) switches to `get_anon_mmap`; the old `std::fs::read` + `Env::get_anon` heap path is gone for the anon FFI. 2. Strip the persistent `LazyConstant` parsed-Constant cache - `LazyConstant.cache` was `Arc<OnceLock<Arc<Constant>>>` and accumulated forever — for mathlib that meant ~30 GB of parsed `Arc<Expr>` trees pinned in the env across the entire run. - New shape: `cache: Option<Arc<Constant>>`. Populated only by `from_constant` (compile-side, where we already own the parsed value). `from_bytes`/`from_mmap_slice` set `None`, and their `get()` parses fresh on every call without storing. - Re-parse cost is bounded by the existing per-worker dedup: `kenv.consts` already addresses each ingressed constant once per work item, and `clear_releasing_memory()` drops the kenv between items. The kenv is now the only persistent materialization layer. 3. `LazyConstant::peek_variant` for `build_anon_work` - One-byte read of the outer Tag4 head to identify the `ConstantInfo` variant — no body parse, no allocation. - `build_anon_work` now dispatches on `peek_variant()`; only `Muts` blocks trigger `lc.get()` (we need the member list for projection-address enumeration), and that `Arc<Constant>` drops at the end of the match arm. - Previously every constant was fully materialized at startup just to read its variant tag. With ~95% of the env being standalone/projection, the work-enumeration pass now skims the env at near-IO speed. Test updates: - `mmap_slice_roundtrips` (already added with the earlier mmap scaffolding) exercises the mmap window roundtrip. - New `from_bytes_does_not_cache` and `from_constant_clones_share_cache` document the new caching contract. - `peek_variant_*` tests cover every variant + empty-bytes and unknown-flag error paths. - `lazy_sparsity_only_materializes_closure` in `src/ix/ixon/env.rs` reframed to assert BFS-of-closure correctness instead of cache side-effects (`is_materialized()` no longer fires for lazy loads). End-to-end on compilemathlib.ixe (--anon, 32 workers): before: 640658/640658 in 266s, peak RSS ~40 GB after: 640658/640658 in 223s, peak RSS 11.3 GB * Address review findings: correctness + small refactors Code-review pass over the anon-mode / mmap / cache-strip work. This commit lands the highest-value subset — the correctness fixes and the small refactors that prevent future drift — and leaves the larger items (format-version bump, sealed marker trait, AnonEnv audit) for separate follow-ups. Correctness: - Verify per-constant address on load (#1). `LazyConstant::verify_address` existed but was never invoked from `Env::get`, `get_anon`, or `get_anon_mmap`. The env-level merkle root catches missing/extra entries but not byte-tampering of a constant whose key is intact; without this check, corruption surfaced much later inside `LazyConstant::get` with a misleading parse error. Inline the `Address::hash(bytes) == addr` check in each loader's Section-2 loop. Added 3 corruption-detection tests (`env_const_bytes_tampering_*`). This required updating a handful of existing tests that stored constants under fake `Address::hash(b"a")` keys instead of their true content hashes — round-tripping such envs now correctly rejects. Added a `store_canonical(env, c) -> Address` test helper for the canonical pattern, and `*_discriminator(refs, n)` so tests can produce content-distinct constants when the same ref-set would otherwise collide. - Hard-error in `ixon_env_to_decoded` on parse failure (#6). `src/ffi/ixon/env.rs` used `filter_map` to silently drop any const whose bytes failed `LazyConstant::get` — the Lean caller had no signal of the lost entries. Switch to a Result-collecting loop and propagate the first parse error; update both FFI call sites (`rs_de_env`, `rs_de_env_anon`). - Doc fixes (#4, #5). `docs/Ixon.md`'s AssumptionTree section described only `Leaf=0x00` and `Node=0x01`, missing `Padding=0x01` (and the docs' `Node` tag collided with the real `Padding` tag — the real `Node` is `0x02`). Also: the env-section table cell said "opt-tagged merkle root" but the implementation writes a bare 32-byte address. Refactors that prevent future drift: - Canonical projection-address helpers (#9). Four production sites reconstructed `Constant::new(IxonCI::{D,I,R,C}Prj{...}).commit().0` by hand; the anon pipeline silently breaks if any one drifts. Extract `defn_proj_address` / `indc_proj_address` / `recr_proj_address` / `ctor_proj_address` (plus `_constant` variants) in `src/ix/ixon/constant.rs`. Update `compile.rs`, `compile/mutual.rs`, and the anon `*_proj_addr` helpers to call them. - `verify_proj_addr_in_env` helper (#21). `ingress_anon_block` had the same "computed-address not in env" check repeated four times (DPrj/RPrj/IPrj/CPrj). DRY into one helper that produces a consistent error format. UX: - Anon display labels switched to `#<hex>` everywhere (#18). Rust was emitting `@<hex>` in progress/fail-out; Lean's `runCheckAnon` was emitting `#{i}` (result index). Standardize on `#<hex>` so the CLI failure summary is joinable with the fail-out file. This required exposing addresses per result slot to Lean — `rsCheckAnonFFI` now returns `Array (String × Option CheckError)` pairing each result with its content-address hex string. Rust builds the pairs via a new `build_anon_result_array` helper. - Stale "check-ixon" header in FailureLog (#16). One-line rename in the doc comment + `# ix check-ixon failures` writeln to `# ix check failures`. Tests: - `testPrimitivesParity` (PrimAddrs regen-parity test). Catches silent drift between hardcoded primitive addresses in `PrimAddrs::new()` and what `rsCompileEnvFFI` produces from the live Lean primitives. Plumbing: new `PrimAddrs::lean_parity_table()` returns `(lean_name, hex)` pairs; new `rs_prim_addrs_canonical` FFI exposes them to Lean; new test in `BuildPrimitives.lean` iterates `kernelPrimitives`, compares against the hardcoded table, and fails with a printable diff on mismatch (with instructions to regenerate). Skipped: `eagerReduce` — synthetic kernel marker whose PrimAddrs value (`0xff…3`) intentionally diverges from its compiled content hash (which collides with `id`). - 3 new corruption-detection tests in `serialize.rs` exercise the Section-2 verify check for `Env::get`, `Env::get_anon`, `Env::get_anon_mmap`. Verification: `cargo test --lib` 1025 passing (3 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe` 105487/105487 in ~21s, `--anon` 89010/89010 with peak RSS 1.4 GB, `.lake/build/bin/IxTests --ignored rust-kernel-build-primitives` shows both `build primitives dump` and `primitive address parity (PrimAddrs vs live compile)` pass. Out of scope (deferred to follow-ups): - #2 Format version bump (Env::FLAG) - #3 rs_kernel_check_consts_anon still uses Env::get on main - #8 Sealed marker trait for the lazy_anon transmute - #11/#12 AnonEnv audit (vestigial wrapper) - #13, #15, #17, #19, #20, #22-25 Various quality cleanups * Round-2 review fixes: mmap defense + small cleanup Six small items from the round-2 review of the anon-mode work: - N1: stat-at-open defense in `Env::get_anon_mmap` (`src/ix/ixon/serialize.rs`). Capture the file size before mmap; if the kernel's mapped length disagrees (file truncated between open and map) bail with a clear error instead of letting workers SIGBUS deep in `LazyConstant::get`. Truncate-in-place under a live mapping is still undefined per POSIX — documented as a caller contract — but the open-time check catches the common case. - New `get_anon_mmap_survives_file_unlink` test: load via mmap, unlink the path, then materialize both already-touched and not-yet-touched constants. Locks in the inode-retention invariant that the SIGBUS analysis depends on; a future refactor that switched to `mmap_anonymous` or copied bytes into a tmpfile would fail loudly here instead of letting workers SIGBUS in production. - #11: delete `AnonEnv::as_ixon_env_unchecked` (`src/ix/kernel/anon_env.rs`). Dead `pub(crate)` escape hatch marked `#[allow(dead_code)]` — confirmed zero callers and removed. - #13: `debug_assert!` on `OnceLock::set` for both result vectors (`src/ffi/kernel.rs`, meta + anon paths). The previous `let _ = results[idx].set(...)` silently dropped a re-set. If a future `build_*_work` dedup refactor breaks the one-write-per-slot invariant, debug builds now panic with the slot index instead of silently losing results. - #19: `allNames.contains` O(n²) preflight → `Std.HashSet` lookup (`Ix/Cli/CheckCmd.lean`). At mathlib scale (~700k env names × thousands of seed names) the previous linear scan-per-name spent measurable seconds on missing-name preflight alone. - N4: doc imprecision in `src/ix/ixon/lazy.rs` — the cache-policy preamble said the worker `KEnv` is "cleared between work items" but the actual cadence is `clear_every` items (`IX_KERNEL_CHECK_CLEAR_EVERY`, default 1 but tunable). Refined wording so the doc matches the code. Verification: `cargo test --lib` 1026 passing (1 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe --anon` 89010/89010 in 15.2s peak RSS 1.4 GB (regression-clean). Out of scope, separate follow-ups: - #2 Format version bump - #3 rs_kernel_check_consts_anon zkPCC FFI (Env::get → mmap) - #8 Sealed marker trait for the lazy_anon transmute - #10 child_arena closure side-effect (widen MField to tuple) - #12 AnonEnv duplicates bfs_refs / transitive_deps_excl - #14 anon worker bypasses M-generic display helpers - #17 Anon --fail-out docstring claims --consts-file compat - #20 Singleton-unwrap duplicated across compile paths - #22-24 Lean `partial def`, `.get!` in tests, unguarded as-u64 casts - N2/N3 Lean is_materialized() callers + compute_const_size_breakdown - Test gaps: rs_kernel_check_anon integration, concurrent mmap, verify_address-on-mmap-corruption, etc. * rustfmt * Tests: derive RawConst addrs from content (verify_address fix) The `Address::hash(bytes) == addr` defense added in f792102 rejects `RawConst { addr, const }` pairs where `addr` is uncorrelated with `serConstant const`. The Rust-side tests in `serialize.rs` were updated at the time (`store_canonical` helper), but three Lean-side sites still produced mismatched pairs and surface now as: × ∃: Env serialization Lean==Rust × ∃: serde RawEnv with data × ∃: serde RawEnv roundtrip deserialization failed: rs_de_env: Env::get: const at idx 0 bytes hash to 6110b739… but stored under b177ec1b… Three fixes, all derive `addr := Address.blake3 (serConstant c)`: - `Tests/Gen/Ixon.lean::genRawConst` — property-test generator was `RawConst.mk <$> genAddress <*> genConstant` (uncorrelated). Now generates the const first, then derives addr from `serConstant`. - `Tests/FFI/Lifecycle.lean::serdeTests` (`withData` unit case) — was using one `testAddr := Address.blake3 #[1,2,3]` for both the const and unrelated blob/comm slots. Split into `testAddr` (still used for the content-hash-free blob/comm) and `testConstAddr := Address.blake3 (serConstant testConst)`. Added a name entry for the new canonical addr. - `Tests/FFI/Lifecycle.lean::genSerdeRawEnv` — the "pool of addresses" pattern picked const addrs from a random pool that couldn't possibly match content hashes. Restructured: consts derive canonical addrs from content, each gets its own name-table entry appended to the pool-derived entries so the serde pipeline's "all addresses resolvable" invariant still holds. * Regenerate Aiur primitive addresses for singleton-unwrap Compiler's singleton non-inductive Muts unwrap (12630aa) changed the content hashes of 48 primitives (`Nat.rec`, `Nat.add`, `Nat.mul`, …). The Aiur kernel's `Ix/IxVM/Kernel/Primitive.lean` still held the pre-unwrap blake3 bytes, so primitive dispatch missed every affected constant and fell through to structural `Nat.rec` unfolding. Tests like `IxVMPrim.nat_mul_big` then drove millions of Nat.succ unfoldings inside the Aiur trace, OOMing the prover. Addresses regenerated from `PrimAddrs::lean_parity_table()` (which the branch had already updated in `src/ix/kernel/primitive.rs`). * Fix standalone Recr ingress: typ-based inductive lookup `find_matching_block_addr` heuristic in the standalone-Recr branch of `build_convert_inputs_walk` picks the inductive block by matching ctor count to rule count. When multiple in-scope inductives share the same ctor count, it returns the wrong block — the stored rules' `ctor_idx` then points to a sibling's ctor, while `populate_rules` (canonical) derives `ctor_idx` from the right inductive's `ctor_indices`. The mismatch surfaces as `compare_rules`'s `assert_eq!(s_ctor, c_ctor)` panic (e.g. `38 != 40` for `IxVMPrim.nat_land_lit`). Switch the standalone-Recr path to the same typ-based resolution `build_aux_recr_ctor_idxs` already uses for aux Recr blocks: peel `params + motives + minors + indices` foralls of `recr.typ`, take the major's head Ref, look it up in `refs` to get the inductive's address, then resolve `IPrj.block` for the Muts wrapper. Slice ctor positions for the specific member via `extract_member_ctor_idxs`. Also store the resolved Muts `block_addr` in `CKRecr` (was passing the heuristic's wrong address through to `convert_recursor`). --------- Co-authored-by: Arthur Paulino <arthurleonardo.ap@gmail.com> Co-authored-by: Samuel Burnham <45365069+samuelburnham@users.noreply.github.com>

plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed).

* Aiur Bytecode → Rust codegen for the IxVM kernel Adds a new Aiur pipeline stage that translates `Bytecode.Toplevel` into a Rust source module, one `fn aiur_fn_N` per Aiur function. The generated code mirrors `src/aiur/execute.rs`'s QueryRecord side effects exactly: same `function_queries.insert` timing, same cache-hit multiplicity bumps, same memory-queries insertion order, same `bytes{1,2}_queries` updates, same IO sequencing. Per-witness trace hashes match the interpreter byte-for-byte on every const tested (`Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). * `Ix/Aiur/Stages/Codegen.lean`: structured Rust IR (`RustExpr` / `RustStmt` / `RustItem` / `MatchArm` / labeled blocks) plus a formatter. Codegen is a structural walk over the Bytecode AST, no string templating. `EmitM = StateM EmitState` threads `(nextVal, nextLabel)` so per-ValIdx Rust locals and per-MC labels are fresh. * Aiur's `map: Vec<G>` is gone in the generated kernel: every ValIdx becomes a Rust local `__v_{i}: G`. Match-arm bodies snapshot `nextVal` on entry so per-arm allocations don't leak to siblings. `MatchContinue` / `Yield` use labeled-block + `break 'label [G; OUT_SIZE]` to bubble yielded values out and rebind them at outer scope. * Deep Aiur recursion uses `stacker::maybe_grow(64 KiB, 4 MiB, …)` at the top of every generated fn, so the native call stack grows on demand rather than pre-reserving a giant thread stack. Verified against shard 24 of the 64-way `init.ixes` partition, which SEGFAULTs without it. * `ix codegen`: new CLI command. Compiles the IxVM Aiur source, walks the bytecode, writes the generated Rust to a fixed path (`src/ix/aiur_ixvm.rs`). The output path is hard-coded; no override. * `src/ix/aiur_ixvm.rs`: the generated kernel, 743 `aiur_fn_*` + `execute_generated` dispatch. Auto-regenerated; do not edit. `#![cfg_attr(rustfmt, rustfmt::skip)]` + a broad `#![allow(unused_*, non_snake_case, clippy::all)]` since the layout is for the compiler, not humans. * `src/ix/aiur_ixvm_runner.rs`: `execute_ixvm(toplevel, fun_idx, args, io_buffer) -> Result<(QueryRecord, Vec<G>), ExecError>`. Same return shape as `Toplevel::execute`, but routes through `execute_generated`. * `src/aiur/synthesis.rs`: `AiurSystem::prove_ixvm(...)` — same shape as `prove`, but the execute step calls `execute_ixvm`. Verification-compatible (proofs from one path verify under the other). * `src/aiur/execute.rs`: helpers exposed for the codegen'd kernel (`bytes{1,2}_execute` → `pub(crate)`, `unconstrained_big_uint_div_mod_helper` extracted, `CodegenBytes{1,2}{Op,}` aliases re-exported, `QueryRecord::new` → `pub(crate)`, `ExecError::InvalidFunIdx` added). * `src/ffi/aiur/protocol.rs`: two new FFI exports — `rs_aiur_toplevel_execute_ixvm` and `rs_aiur_system_prove_ixvm`. Same wire format as the existing `_execute` / `prove` exports. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.executeIxVM` — mirror of `execute`. * `Ix/Aiur/Protocol.lean`: `AiurSystem.proveIxVM`. * `Ix/Cli/CheckCmd.lean`: `runCompiled` now dispatches through `executeIxVM`. The Rust bytecode interpreter is no longer reachable from `ix check` (the Lean-side `--interp` fallback stays for richer error diagnostics). * `Ix/Cli/ProveCmd.lean`: `proveOne` uses `proveIxVM`. * `Cargo.toml`: `stacker = "0.1"`. For witnesses where execute is the dominant work (single-const checks via `ix check 'X'`), the codegen'd kernel is ~1.6× faster than the bytecode interpreter end-to-end (measured on `Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). Tiny consts are within ~10% (stacker probe overhead amortises). Peak RSS is within a few MB of the interpreter — stacker grows the stack on demand, no giant upfront reservation. For shard-driven `ix check --shard K` runs the speedup is invisible: per-shard wall time is dominated (~92%) by `buildShardCheckEnvWitness` in Lean, NOT by the kernel execute (~8%). Cutting witness construction is the next lever. `Nat.add_comm` proof produced via `proveIxVM` verifies under the existing `AiurSystem::verify`. End-to-end: lake exe ix prove 'Nat.add_comm' → proof addr 0d0ab0f9… lake exe ix verify 0d0ab0f9… → ok * Move shard witness construction to Rust + parallelise Replaces `IxVM.ClaimHarness.buildShardCheckEnvWitness` (Lean side, ~92% of shard wall time on heavy partitions) with a Rust port that builds the `aiur::execute::IOBuffer` directly, without per-byte boxing into Lean `Aiur.G` values. The two hot phases run on rayon: * Closure walk: each owned addr's transitive `Constant.refs` + projection-block traversal runs on its own thread; results are deduped through a `dashmap::DashSet`. * Byte-to-G conversion: per-const `(key, data)` tuples are built in parallel chunks of 256; final IOBuffer assembly (channel arena append + key→`IOKeyInfo` map insert) runs serially because the arena index is monotonic. * `src/ix/aiur_ixvm_witness.rs` (new): `build_shard_check_env_witness` produces `(claim, claim_digest_input, io_buffer)` ready to feed to `execute_ixvm`. Mirrors the 6-channel layout documented in `Ix/IxVM/ClaimHarness.lean` (claim / asm tree / const bytes / Defn hint / blob discriminator / blob raw bytes). * `src/ffi/aiur/protocol.rs`: new FFI `rs_aiur_toplevel_shard_check_ixvm` bundles witness build + `execute_ixvm` into one cross-language trip. Returns the same `(output, ioBuffer, queryCounts)` shape as `rs_aiur_toplevel_execute_ixvm`, so the Lean shim stays drop-in compatible. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.shardCheckIxVM` wraps the FFI. * `Ix/Cli/CheckCmd.lean`: shard-mode `ix check` (`--ixe + --ixes`) now dispatches through `runShardOwnedNative` → `shardCheckIxVM`, bypassing the Lean witness builder. Single- shard and whole-partition paths share the fast route; the legacy `runShardCheckManifest` / `runShardCheckAll` callbacks stay live for `--interp` only. Shard 26 of the 64-way `init.ixes` partition, on the same host: | Variant | total | speedup | | --- | --: | --: | | Lean witness + bytecode-interp (baseline)| 1029 s | 1.0× | | Lean witness + codegen kernel | 1015 s | 1.01× | | Rust witness (serial) + codegen | 101 s | 9.95× | | Rust witness (parallel) + codegen | 80 s | 12.9× | The serial Rust witness collapses the 935 s of Lean per-byte boxing into ~23 s; rayon hides that 23 s under the codegen-kernel execute (~78 s) so witness build effectively becomes free at the shard level. FFT cost matches the bytecode interpreter exactly (107_006_963_281), so the QueryRecord layout is bit-identical across all four variants. * Closure walk uses single-source BFS with the global visited set shared via `DashSet::contains` checks before pushing to the per-thread stack — avoids redundant work without a per-owned union step. * Per-channel ordering: each chunk's partial `ChannelEntries` feeds the serial fold in iteration order, so channel arena contents match the Lean side's exactly. Test still relies on trace-hash parity from the earlier codegen commit, which exercised the same path through the bytecode interpreter and the codegen kernel. * Coverage check is currently skipped on the IxVM-native whole-partition path (`runShardManifestAllNative`); legacy `runShardCheckAll` still does it for `--interp`. Re-enable separately if needed. * Codegen: value-returning byte ops, dead-fold the unconstrained plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed). * Wire Rust witness fast path through every check/prove entry point Per-claim mode (a `Claim.check addr none`) builds a closure rooted at the target address. That closure can be the whole environment when the target is a heavily-shared root constant. The Lean witness builder paid per-byte boxing into `Aiur.G` for every byte in that closure — the same cost we already eliminated for shard mode. # Surface Five new FFIs, all bundling witness build + execute_ixvm (and STARK prove where applicable) into one cross-language trip so the `IOBuffer` never crosses the boundary mid-pipeline: * `rs_aiur_toplevel_check_addr_ixvm` — `Bytecode.Toplevel.checkAddrIxVM (toplevel) (funIdx) (ixePath) (addrBytes)`: per-claim check; builds witness for `Claim.check addr none` via `build_claim_check_witness`, runs `execute_ixvm`. Same return shape as `shardCheckIxVM`. * `rs_aiur_toplevel_check_env_bytes_ixvm` — same as above but takes the env as a serialized byte blob (`Ixon.serEnv`) instead of a `.ixe` path. Used by the compiled-Lean-env code path (`ix check NAME` without `--ixe`), where the env is built in Lean memory. * `rs_aiur_system_prove_addr_ixvm` — `AiurSystem.proveAddrIxVM (...)`: per-claim prove (witness + execute + STARK prove all in Rust). * `rs_aiur_system_prove_env_bytes_ixvm` — bytes-blob counterpart. * `rs_aiur_system_shard_prove_ixvm` — `AiurSystem.shardProveIxVM (...)`: per-shard prove end-to-end in Rust. `build_claim_check_witness` lives next to `build_shard_check_env_witness` in `src/ix/aiur_ixvm_witness.rs` and uses the same parallel closure walk + parallel byte→G conversion. The bytes-blob FFIs additionally harvest `anon_hints` from each `Def` named entry after `Env::get` — `Env::get` (full form, used by the bytes-blob path) doesn't populate `env.anon_hints` the way `Env::get_anon` does, but the kernel's `verify_claim` reads ch 3 (Defn reducibility hints) so the hints must end up in the env. This mirrors the harvest pass already inside `get_anon` at `src/ix/ixon/serialize.rs:1683`. # Wiring `Ix.Cli.CheckCmd.WitnessSource`: a three-arm discriminated union threaded through `forEachClaim` (the shared check/prove driver): * `.native ixePath addr` — `.ixe`-backed env, Rust mmap. * `.nativeBytes envBytes addr` — Lean-memory env serialized to bytes, Rust decodes via `Env::get`. * `.lean witness` — pre-built `ClaimWitness` (fallback). `runCompiled` and `proveOne` dispatch on the source. # Coverage | Command | Path | |---------------------------------------|-----------------------------------------------| | `ix check NAME --ixe` | **`checkAddrIxVM`** (new) | | `ix check NAME` (no `--ixe`) | **`checkEnvBytesIxVM`** (new) | | `ix check --claim hex --ixe` | `checkAddrIxVM` for `check addr none` | | `ix check --ixes --shard K` | `shardCheckIxVM` (existed) | | `ix check --ixes` | `shardCheckIxVM` (existed) | | `ix prove NAME --ixe` | **`proveAddrIxVM`** (new) | | `ix prove NAME` (no `--ixe`) | **`proveEnvBytesIxVM`** (new) | | `ix prove --ixes --shard K` | **`shardProveIxVM`** (new) | | `ix prove --ixes` | **`shardProveIxVM`** loop (new) | | `Benchmarks/Typecheck.lean` Phase 1 | `checkAddrIxVM` / `executeIxVM` | | `Benchmarks/Typecheck.lean` Phase 2 | `proveAddrIxVM` / `proveIxVM` | | `Benchmarks/IxVM.lean` | `proveIxVM` (was `prove`) | `--interp` route is preserved: it materialises any `WitnessSource` back into a `ClaimWitness` before driving the Aiur source interpreter. `--claim <hex>` over non-`check addr none` variants (`eval` / `reveal` / `contains` / `checkEnv-with-asm`) still uses the Lean witness builder. # Sanity * `ix check Nat.add_comm --ixe init.ixe` (warm): 3.1 s wall, FFT = 23_603_449. * `ix check Nat.add_comm` (compiled env via bytes blob, warm): 2.4 s wall, FFT = 23_603_449. # Known overheads (tracked for a follow-up redesign) * **Per-claim env re-parse on `--ixe + many names`**: each FFI call re-runs `Env::get_anon_mmap` on the same path. Mathlib-scale iteration pays the lazy-index build N times instead of once. * **`--interp + .nativeBytes` round-trip**: Lean serializes the compiled env then `materialise` deserializes it back, just to feed `runInterp`. * **`runShardProveNative` claim reconstruct**: redoes the closure walk + canonical `AssumptionTree` build Lean-side after `shardProveIxVM` already computed the same claim internally. A future `EnvHandle`-based API would close all three: build the env once into a Rust-owned handle, pass the handle to every per-claim/per-shard FFI, and have prove FFIs return the serialized claim bytes. * EnvHandle: Rust-owned env shared across per-claim / per-shard FFI calls Before: every per-claim and per-shard FFI re-parsed the Ixon env (`Env::get_anon_mmap` per call). On `--ixe + many names` / all-shards prove, this is the dominant overhead for iteration-heavy workflows — O(num_consts) lazy-index build × N targets. After: the env lives once per CLI invocation in a Rust-owned `EnvHandle`. Lean holds an opaque `Aiur.EnvHandle` reference and threads `@& EnvHandle` through every per-target FFI call. The env is parsed exactly once at handle construction; downstream calls share it. # Surface Five new FFIs collapse the previous six per-call variants (`checkAddrIxVM`, `checkEnvBytesIxVM`, `shardCheckIxVM`, `proveAddrIxVM`, `proveEnvBytesIxVM`, `shardProveIxVM` — all deleted): * `rs_aiur_env_handle_from_ixe(path) → LeanExternal<EnvHandle>`: mmap-load via `Env::get_anon_mmap`. Anon parser already harvests `anon_hints`; no post-pass. * `rs_aiur_env_handle_from_bytes(blob) → LeanExternal<EnvHandle>`: decode `Ixon.serEnv`-shape blob via `Env::get` + harvest `anon_hints` from each `Def` named entry. Used by the compiled-Lean-env path. * `rs_aiur_toplevel_check_addr_with_env(toplevel, fun_idx, handle, addr_bytes)`: per-claim check. Reuses handle's parsed env. * `rs_aiur_toplevel_shard_check_with_env(toplevel, fun_idx, handle, owned_blob)`: per-shard check. * `rs_aiur_system_prove_addr_with_env(system, fri, fun_idx, handle, addr_bytes) → (claim_bytes, proof, ioBuffer)`: per-claim prove. Rust serializes the reconstructed `Ix.Claim` via `ixon::Claim::put` so Lean can deserialize via `Ixon.runGet Ix.Claim.get` directly — no closure walk + canonical `AssumptionTree` recomputation Lean-side. * `rs_aiur_system_shard_prove_with_env(system, fri, fun_idx, handle, owned_blob) → (claim_bytes, proof, ioBuffer)`: per-shard prove. # Lean * `Aiur.EnvHandle` opaque type with `fromIxe` / `fromBytes`. * `Aiur.Bytecode.Toplevel.checkAddrWithEnv` / `shardCheckWithEnv`, `Aiur.AiurSystem.proveAddrWithEnv` / `shardProveWithEnv`. * `Ix.Cli.CheckCmd.Target` replaces `WitnessSource`: ```lean inductive Target where | addr (a : Address) -- Claim.check addr none | shard (owned : Array Address) -- Claim.checkEnv | leanW (w : ClaimWitness) -- --interp, --claim non-check ``` `runCompiled` / `proveOne` take `(envHandle?, target)`. The envHandle is `none` only for `.leanW` (`--interp` legacy path). * `runShardOwnedNative` and the manifest-driver helpers take the envHandle as a parameter. Single-shard mode builds it once; all-shards mode reuses the same handle across every shard's FFI call (eliminates per-shard re-mmap). * `runShardProveNative` deserializes the wire claim bytes via `Ixon.runGet Ix.Claim.get` instead of re-running `shardCheckEnvClaim`. # Benchmarks `Benchmarks/Typecheck.lean` builds the envHandle once before Phase 1, reuses it across Phase 1 (execute) + Phase 2 (prove). Both phases now go through `checkAddrWithEnv` / `proveAddrWithEnv` on the full-closure path. `--subject-only` still uses Lean `buildVerifyConst` + `executeIxVM` / `proveIxVM` (witnesses intentionally small there). # New crate `crates/ix/src/env_handle.rs` — the `EnvHandle` struct + constructors live in the existing `ix` crate next to the witness builder and codegen runner. # Sanity * Multi-target check (warm): `ix check --ixe init.ixe Nat.add_comm Nat.add` → 3.3 s wall, single envHandle shared across both targets. * Shard 26 check (warm): `ix check --ixe init.ixe --ixes init.ixes --shard 26` → ~78 s wall, FFT = 107_006_963_281 (parity preserved). # Coverage | Command | Path | |--------------------------------------|-----------------------------------| | `ix check NAME --ixe` | `checkAddrWithEnv` | | `ix check NAME` (no `--ixe`) | `checkAddrWithEnv` + fromBytes | | `ix check --claim hex --ixe` | `checkAddrWithEnv` for check-none | | `ix check --ixes --shard K` | `shardCheckWithEnv` | | `ix check --ixes` | `shardCheckWithEnv` × all shards | | `ix prove NAME --ixe` | `proveAddrWithEnv` | | `ix prove --ixes --shard K` | `shardProveWithEnv` | | `ix prove --ixes` | `shardProveWithEnv` × all shards | | `Benchmarks/Typecheck` Phase 1+2 | `checkAddrWithEnv` / `proveAddrWithEnv` | Only `--interp` and `--claim hex` over a non-`check addr none` persisted claim still build a Lean `ClaimWitness`; both go via the `.leanW` target arm. * ix codegen: add --check flag for CI; gate stale generated kernel `ix codegen --check` compares the emitted Rust source against the on-disk `crates/ix/src/aiur_ixvm.rs` and exits 0 if identical, 1 otherwise. No write side effect. ~2 s warm; fast enough to gate on every PR. Wired into the `lean-test` CI job so a forgotten regen on a kernel-touching PR fails CI instead of merging stale generated code that drifts from the Bytecode → Rust emitter. * ix check: --interp {source|bytecode} to bypass codegen kernel Two interpreter modes behind a single `--interp` flag: * `--interp source`: Aiur source interpreter (`Aiur.runFunction` over the source-level `Decls`). Richer per-step error diagnostics. * `--interp bytecode`: generic Aiur bytecode interpreter (`Bytecode.Toplevel.execute`, `rs_aiur_toplevel_execute` route). Skips the `ix codegen` + `cargo build --release` cycle needed after editing `Ix/IxVM/*.lean` — the bytecode is rebuilt Lean-side at exe load. Slower per-check than the codegen kernel; ideal for tight iteration on the IxVM source. * omit the flag entirely for the native codegen kernel (default). Invalid values (e.g. `--interp foo`) fail fast with a clear error. # Plumbing The `check_addr_with_env` and `shard_check_with_env` FFIs take a `use_bytecode: bool` and dispatch via a shared `dispatch_execute` helper. `--interp bytecode` sets it to `true`. The `.leanW` arm of `runCompiled` also picks between `Bytecode.Toplevel.execute` and `executeIxVM` based on the same flag. The `--interp source` path requires a Lean `ClaimWitness`. The existing driver had switched to `.addr`/`.shard` targets against the Rust-owned `EnvHandle`; source-interp had no way to materialise those. `forEachClaim` now takes a `forceLeanWitness : Bool` — when true, it builds a Lean witness for every target (via `mkWitness` / `loadIxonEnv` for the compiled-Lean-env path) and passes `.leanW` so `runInterp` can consume it directly. # Sanity * `ix check Std.Time.Week.Offset.ofMilliseconds` (warm): 9.3 s wall, FFT = 12_430_516_949 (codegen kernel). * `ix check --interp bytecode Std.Time.Week.Offset.ofMilliseconds` (warm): 14.6 s wall, FFT = 12_430_516_949 (bytecode interp). * `ix check --interp source Eq` (warm): 6.3 s wall, output `Eq: ()`. * `ix check --interp garbage Nat.add_comm`: exits 1 with clear error. * IxVM codegen: regen + rebase-fixups after KLevel port Aligns the ap/codegen-ixvm-native stack with main's KLevel pointer port (b84a500) and re-lands clippy-clean under the workspace lint set: - Regen crates/ix/src/aiur_ixvm.rs from Aiur source (KLevel = &KLevelNode changes the generated fn shapes; ~26k lines net churn). - Extend the generated file's #![allow(...)] header (via Ix/Aiur/Stages/Codegen.lean) with clippy::ptr_as_ptr, clippy::match_same_arms, and clippy::large_types_passed_by_value — three lints the codegen's straight-line style trips deterministically on every regen and that clippy::all doesn't cover. - rustfmt reflow in crates/aiur/src/execute.rs and a handful of ix / ffi files that CI now enforces via the ap/aiur-aux-recursor-parity toolchain. - Fix 4 real clippy warnings uncovered under the workspace lint set: * env_handle.rs: clone-on-Copy → deref (ReducibilityHints is Copy). * aiur_ixvm_witness.rs: collapse if-let-if, drop needless continue, slice::from_ref instead of &[x.clone()]. * aiur_ixvm_runner.rs: keep execute_ixvm's Vec<G> arg (required by AiurSystem::prove_ixvm's fn-pointer bound) + local #[allow( clippy::needless_pass_by_value)] with a comment explaining why. * ffi/aiur/protocol.rs: unqualify aiur::G/IOBuffer/QueryRecord after the workspace lint added unused_qualifications; is_multiple_of() over % 0. Verified: cargo clippy --workspace --all-targets --all-features --D warnings clean; lake test -- --ignored ixvm passes. * ix-check + ixvm tests: coverage gate, --interp source fix, codegen parity + rename ix crate to ixvm-codegen Four review fixes on the codegen-ixvm-native stack. - **Coverage gate on \`--ixes\` (all-shards, native path).** \`runShardManifestAllNative\` now calls \`shardsCover\` before running any shard, matching the pre-refactor \`runShardCheckAll\` behavior. Without this, a stale/truncated \`.ixes\` manifest makes \`ix check --ixes\` exit 0 while some env constants are never checked by any shard — the mundane trigger being: edit a definition, rebuild the \`.ixe\`, forget to re-run \`ix shard\`, and the constants that go unchecked are exactly the ones just edited. Single-shard \`--shard K\` path is unchanged (consistent with pre-refactor). \`shardsCover\` moved above \`runShardManifestAllNative\` to satisfy forward-decl ordering. - **\`ix check --ixe … --claim <hex> --interp source\` regression.** The \`claimHex\` arm in \`forEachClaim\` ignored \`forceLeanWitness\` and always mapped \`.check addr none\` claims to \`Target.addr\`. Under \`--interp source\`, \`runInterp\` then rejects \`.addr\` with "\`--interp requires a Lean witness; addr/shard targets unreachable here\`". Now mirrors the sibling name-based arms and routes through the Lean witness path when \`forceLeanWitness\` is on. Only bit the common claim shape. - **Codegen ↔ bytecode parity CI test.** New \`runParityCase\` + \`parityCases\` in \`Tests/Ix/IxVM.lean\` wire every constant already listed in \`kernelCheckEntries\` plus \`kernel_unit_tests\` through BOTH \`Toplevel.execute\` (bytecode) and \`Toplevel.executeIxVM\` (codegen'd Rust), and diff the returned \`(output, IOBuffer, QueryCounts)\` triples. Turns "generated kernel ≡ interpreter on QueryRecord" from reviewed-by-hand into checked-by-CI. 129 assertions run per suite invocation (43 constants × output + IOBuffer + QueryCount), all pass on this branch head. - **Rename \`crates/ix\` → \`crates/ixvm-codegen\`.** The old \`ix\` crate held only the codegen'd IxVM kernel + its runner + witness helpers. \`ix-kernel\` is the hand-written Rust ix typechecker; \`ix-common\` / \`ix-compile\` are Ix-level infrastructure. \`ixvm-codegen\` cleanly names what this crate holds. Path updates: workspace \`members\` and \`workspace.dependencies\`, ffi crate dep + \`use\` sites, Lean \`codegenOutPath\` in \`Ix/Cli/CodegenCmd.lean\`, doc reference in \`Ix/Aiur/Semantics/BytecodeFfi.lean\`. \`ix codegen\` now writes to \`crates/ixvm-codegen/src/aiur_ixvm.rs\`. Verified: \`cargo check --workspace\` + \`lake test -- --ignored ixvm\` both exit 0 post-rename; parity + pinned FFT tests all pass. * cargo-deny: allow ar_archive_writer's Apache-2.0 WITH LLVM-exception Post-rename, ixvm-codegen pulls stacker → psm → ar_archive_writer (a build-time transitive). ar_archive_writer is licensed under the LLVM-exception variant of Apache-2.0 (standard for LLVM tooling; no additional runtime obligation for downstream users), which isn't on the workspace allow list. Add a per-crate exception scoped to ar_archive_writer so we don't blanket-allow the license family across the tree.

arthurpaulino · 2025-02-04T13:52:08Z

No description provided.

Lays the groundwork for measuring the kernel performance plan (plans/okay-let-s-write-a-lucky-dolphin.md) per audit §10. Counters live in a new kernel::perf module; KEnv carries a PerfCounters field that is dumped from a Drop impl when IX_PERF_COUNTERS=1 is set. Unset is the production default — every increment short-circuits via a LazyLock<bool> so the cost is a single cached branch on the hot path. Wired into the cache get sites the audit identified: - whnf_cache hits/misses (whnf.rs around 201/211) - whnf_no_delta_cache hits/misses (whnf.rs around 437/446) - infer_cache and infer_only_cache hits/misses (infer.rs around 45/51) - def_eq_cache hits/misses (def_eq.rs around 137/149) - def_eq_failure set hits/inserts (def_eq.rs around 360) - per-constant peak/avg MAX_REC_FUEL consumption (recorded in TypeChecker::reset before the next constant resets it) Verified against lake test -- kernel-check-env --ignored: 192156/192156 constants pass in 251s (4% over the 242s baseline; overhead is the LazyLock branch + atomic load in the disabled path). P1: def_rank_id replaces def_weight_id for hint-priority comparison Audit Tier 1 #3 (kernel-perf-adversarial-audit-2026-04-26.md, §4.2): the prior u32 encoding mapped Abbrev to u32::MAX-1 and saturating-added Regular(h) to h+1, which collide at h ≥ u32::MAX-2. When that happens the delta-direction logic treats Abbrev and a maximally-heavy Regular as "same height" and unfolds both, instead of preferring Abbrev as Lean does (compare(d_t->get_hints(), d_s->get_hints()) at type_checker.cpp:910). Replace the u32 weight with a (class: u8, height: u32) tuple compared lexicographically: - Opaque / Theorem / unknown → (0, 0) - Regular(h) → (1, h) (height ordering preserved within class) - Abbrev → (2, 0) (strictly above every Regular) Update the two call sites (is_def_eq lazy-delta height comparison and lazy_delta_step). The map_or default for "missing head" is preserved as (u8::MAX, u32::MAX) — the branch is dead in practice (a_delta && b_delta imply both heads are present) but kept consistent with the prior u32::MAX sentinel. Two regression tests: - def_rank_abbrev_above_saturated_regular: Abbrev outranks Regular(u32::MAX) (the previous saturation collision). - def_rank_regular_orders_by_height: height monotonically orders Regular ranks within the class. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 256s (no regression vs the 242s pre-perf-counters baseline; the +14s is from the IX_PERF_COUNTERS=unset LazyLock branch added in the prior commit, not this change). P2: peel_proj_forall fast-paths syntactic Pi in projection inference Audit Tier 1 #2 (kernel-perf-adversarial-audit-2026-04-26.md, §7.2): infer_proj's two parameter-consuming loops (param peel and field peel) called self.whnf(&r)? unconditionally per iteration, on a body mutated by subst at the previous step. The whnf cache rarely hits between iterations and each call re-traverses the substituted body. Extract a peel_proj_forall(&r, err) helper that: - tries ExprData::All(..) syntactically first (no WHNF call), and - falls back to full self.whnf(e) only when the binder isn't already syntactic Pi. This mirrors Lean's inferProj at type_checker.cpp:582–610. Both projection-inference loops now call peel_proj_forall instead of unconditional whnf. Behaviorally equivalent — same WHNF semantics on miss, no semantic change otherwise. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 258s (parity with the post-pre-work, post-P1 baseline; no measurable regression and the cache-hit-rate counters will move on tactic-heavy workloads under IX_PERF_COUNTERS=1). P3a: WhnfFlags substrate (no behavior change) Lays the foundation for the Lean4Lean architectural alignment described in plans/okay-let-s-write-a-lucky-dolphin.md. Phase 3a is substrate-only — no call site is migrated to cheap mode yet, so behavior is unchanged. Adds: - WhnfFlags { cheap_rec, cheap_proj } with FULL and CHEAP consts and is_full(). CHEAP is currently equal to FULL until Phase 3c wires it. - whnf_core_with_flags (private): the existing whnf_core impl, now threading flags into recursive calls and try_iota_with_flags. - whnf_core / whnf_core_cheap (super): FULL/CHEAP wrappers. - whnf_no_delta_with_flags (private): the existing whnf_no_delta impl with the Prj branch gated on cheap_proj — falls back to full whnf on the projected value when not cheap. - whnf_no_delta (pub) / whnf_no_delta_cheap (super): wrappers. - try_iota_with_flags: gates major-premise WHNF and string-literal constructor reduction on cheap_rec. - try_proj_app_reduce_with_flags: gates projected-value WHNF on cheap_proj. Cache reads/writes (whnf_no_delta_cache, equiv-manager second-chance) are gated on flags.is_full(): cheap callers neither read nor write the cache, preserving the invariant that any cached entry is a fully-reduced normal form. Phase 3b will inline the projection branch into whnf_core to match Lean4Lean's two-layer architecture (refs/lean4lean/Lean4Lean/ TypeChecker.lean:266, 297). Phase 3c will flip CHEAP to enable cheap_proj and migrate specific def-eq sites. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 243s (matches the 242s baseline; substrate adds no measurable overhead when CHEAP == FULL). P3b: inline projection into whnf_core (Lean4Lean architectural alignment) Move the Prj branch from whnf_no_delta_with_flags into whnf_core_with_flags so our whnf_core matches Lean4Lean's whnfCore semantics exactly (refs/lean4lean/Lean4Lean/TypeChecker.lean:284-292, 337-341). Before this commit: whnf_core — beta + zeta + iota + cheap projection (recursive whnf_core on val, no delta) whnf_no_delta — whnf_core + FULL projection (full whnf on val) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta Lean4Lean's architecture has no whnf_no_delta layer. Their whnfCore includes projection, with the cheap_proj flag deciding whether the projected value uses whnfCore (cheap) or whnf (full). After this commit: whnf_core (with WhnfFlags) — beta + zeta + iota + projection (cheap_proj controls val reduction) whnf_no_delta — whnf_core(_, FULL) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta The bare-Prj branch in whnf_no_delta_with_flags is removed — whnf_core now handles it directly. The App-of-Prj branch stays in whnf_no_delta because whnf_core's loop returns once the outermost Prj is resolved; try_proj_app_reduce_with_flags gives one more attempt at the same cheap_proj policy when the outer expression is App(Prj, ...). Pure refactor, no semantic change with CHEAP == FULL. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 270s (within noise of the 243s pre-refactor baseline). Phase 3c will flip CHEAP to enable cheap_proj=true and migrate def-eq's lazy-delta sites surgically per Lean4Lean's pattern. P3c (postponed): document the HeaderParsedSnapshot regression P3a (substrate) and P3b (Lean4Lean architectural alignment) are committed. Phase 3c — flipping CHEAP to enable cheap_proj=true and migrating the def-eq lazy-delta sites — was attempted but reproduced 5 failures on chained projections in Lean.Language.Lean.HeaderParsedSnapshot.* even after P3b inlined the projection branch into whnf_core. The substrate is left in place; CHEAP stays equal to FULL until the regression's root cause is understood. Notes on the regression for the next investigator: - Failures: HeaderParsedSnapshot.{stx,result?,metaSnap,toSnapshot,ictx}, all with 'projection: type mismatch with declared struct'. - The struct `extends` a parent, so each projection is a chained Prj whose val is itself a Prj into the parent. - The error comes from infer.rs's infer_proj at the head-vs-struct_id address compare, after FULL whnf on val_ty. That whnf is FULL, but val_ty was inferred via paths that may have consulted a def-eq cache populated under cheap mode. Possible cache-poisoning suspect: def_eq_cache writes a `false` result for inputs whose lazy-delta loop bottomed out under cheap projections. The cache key uses raw a/b hashes, not cheap-reduced shapes, so a stored `false` is indistinguishable from a FULL `false` by future readers. - Lean4Lean does not have an analogous wide def_eq_cache; their failure cache is keyed only on same-spine pairs in lazyDeltaReductionStep. Future P3c iterations should either prove the cache poisoning theory incorrect or restrict def_eq_cache writes to FULL-derived results.

* Refactor Claim ADT + add merkle infrastructure for ZK aggregation Adds the format hooks needed before recursive verification lands in Aiur. Five-variant Claim ADT with explicit assumption commitments, a canonical Blake3 merkle module, a serializable AssumptionTree for recovering leaf sets, and a Contains claim for the discharge step. Claim ADT (5 variants): - Eval { input, output, assumptions: Option<Address> } - Check { const_addr, assumptions: Option<Address> } - CheckEnv { root, assumptions: Option<Address> } - Reveal { comm, info } -- unchanged, no assumptions - Contains { tree, const_addr } -- new, for inclusion proofs Tag4 reorganized to keep everything in single-byte tags: - 0xE for env, comm, AssumptionTree, and claims (slots 0-7) - 0xF for proofs (slots 0-4; 5-7 reserved) - Comm moved from variant 5 -> 1 Matches the "Variant (0-7)" constraint documented in docs/Ixon.md. Env serialization: - Every .ixe file now carries a canonical merkle root over its consts.keys() in the on-disk header (non-optional, 32 bytes; empty const sets use the zero-address sentinel). - Two envs with the same const set produce byte-identical roots regardless of insertion order. Verified on deserialize. New modules: - src/ix/ixon/merkle.rs + Ix/Merkle.lean: canonical sorted builder, free-form merkle_join composition, membership proofs, domain separation per RFC 6962. - src/ix/ixon/assumption_tree.rs + Ix/AssumptionTree.lean: serializable merkle tree with Leaf/Padding/Node variants. canonical() builds the same shape merkle_root_canonical hashes; join() is O(1) free-form composition. - src/ix/kernel/claim.rs: builders that compute transitive-dep assumptions from an env (build_check_claim, build_eval_claim, build_check_env_claim, env_merkle_root). Other: - Extracted shared BFS walker on Env::bfs_refs + Env::transitive_deps_excl; the inlined test-feature copy in lean_env.rs now calls into it. - Lean FFI export rs_env_merkle_root for cross-impl verification of the env root. - Proof bytes for all variants are uniform opaque ZK bytes; witness data (e.g., Contains merkle paths) is prover-side scratch consumed by the ZK circuit and not transmitted on the wire. - docs/Ixon.md Tag4 tables and env section updated; .ixe extension documented. Recursive verification (the ZK proof generation for Contains and the aggregation discharge transitions) is intentionally deferred to a follow-up. Tests: 993 Rust unit tests pass (was 953 pre-refactor), 813 Lean tests pass with no failures; cargo clippy clean. * Add lazy env deserialization and anon-mode kernel FFI Two related changes to reduce kernel memory + lay groundwork for metadata-isolated typechecking. **Lazy constant deserialization.** `Env::consts` now stores `LazyConstant` (`Arc<[u8]>` + `OnceLock<Arc<Constant>>`) instead of `Constant`. Constants are materialized on first access and the structured form is cached. Sparse access patterns (single-constant typecheck, transitive_deps walk, claim builders) only parse the closure they need; non-reachable constants stay as raw bytes. The `.ixe` section-2 layout gains a Tag0 length sidecar before each constant's Tag4 bytes: old: [addr:32] [Tag4 constant bytes] new: [addr:32] [Tag0 length] [Tag4 constant bytes] The length is section-level framing, not part of the constant's content hash. `Address::hash(raw_bytes) == addr` is preserved. `Constant::put`/`Constant::get` are unchanged. The Lean side (`Ix.Ixon.putEnv`/`getEnv`) reads/writes the new format. **Anon-mode kernel FFI.** New `rs_kernel_check_consts_anon(path, addrs, quiet)` exposes anonymous-mode typechecking by content address. The kernel runs as `KEnv<Anon>` / `TypeChecker<Anon>` with every `M::MField<T>` erased to `()`, so the typechecking logic structurally cannot read metadata. Useful for zkPCC verifiers that hold only addresses. Supporting pieces: - `KernelMode::HAS_META: bool` for future compile-time gating. - `AnonEnv<'a>` wrapper (`src/ix/kernel/anon_env.rs`) exposing only consts/blobs/transitive walks — no `named`/`names`/`comms`. - `Env::get_anon` reads header + blobs + consts, parse-and-drops metadata sections (3-5), returns an `Env` with empty `named`/ `names`/`comms`. Same merkle-root verification as `Env::get`. - `rs_de_env_anon` FFI + `Ix.Ixon.rsDeEnvAnon` Lean wrapper. - `Ix.KernelCheck.rsCheckConstsAnonFFI` Lean binding. Caveat: `ixon_ingress::<Anon>` still consults `Env::named` internally to enumerate work items. The resulting `KEnv<Anon>` is metadata-free so the typechecker is anon, but full ingress-level metadata isolation is a follow-up. Tests: 16 new (9 lazy + sparsity; 3 AnonEnv; 4 get_anon). All 1009 Rust unit tests pass; all 813 Lean tests pass; clippy clean. * Simplify ix check + add metadata-free anon mode - `ix check` becomes .ixe-only with a positional `<path>`. Drops the `--lean` compile-and-check flow and the `--env` flag; direct Lean → kernel checks remain available through `rsCheckConstsFFI` for tests. Removes the redundant `CheckIxonCmd.lean`. - New `--anon` flag runs a metadata-free kernel check: loads the .ixe via `Env::get_anon` (discards named/names/comms), enumerates work items from `env.consts` alone, and rebuilds member + ctor projection addresses deterministically via `Constant::commit`. Rejects `--consts`/`--ns`/`--consts-file` since it always checks everything. - Compiler: unwrap singleton non-inductive Muts blocks to standalone Defn/Recr. `Expr::Rec(0, univs)` already resolves correctly against a one-member block, so the wrapper was pure overhead; this keeps the env structurally uniform and matches `compile_single_def`. The anon ingress for standalones passes `mut_ctx_override = [self_id]` so self-recursive standalones still typecheck. - Kernel: anon parallel runner mirrors `run_checks_parallel_on_large _stacks` (32 workers, slow-detection, RSS tracking). Genericized `check_one_const<M>` / `check_consts_loop<M>` / `format_tc_error<M>` to share the runner. `KernelMode::meta_field_with` / `meta_field_try` gate metadata lookups in expression ingress at compile time. - Anon lazy ingress (`LazyAnonIngress` in tc.rs, helpers in ingress.rs) dedupes blocks via `kenv.blocks.contains_key(&KId<Anon>(B, ()))` before re-running `ingress_anon_block` (prevents N² re-ingestion when sibling projections fault separately within one worker check). - `Env::get_anon` now harvests `ReducibilityHints` from the otherwise-discarded Named section into a sidecar `Env::anon_hints: FxHashMap<Address, ReducibilityHints>`. Anon ingress threads these through a new `hints_override` parameter on `ingress_defn` so the lazy-delta tiebreak in `def_eq::def_rank_id` sees realistic heights instead of `Regular(0)`. Hints are performance advice, not correctness data — supplying them in anon mode preserves the metadata-free trust model. - Regenerate the canonical addresses in `PrimAddrs::new()`; singleton- unwrap changed the content hash of every self-recursive primitive (Nat.add, String, Nat.rec, …). End-to-end on compileinitstd.ixe (105,487 constants): meta: 105487/105487 in 20.6s, peak RSS 7.4GB anon: 89010/89010 in 17.8s, peak RSS 4.0GB * Clean up cargo clippy --all-targets - 11× `cloned_ref_to_slice_refs`: `&[x.clone()]` → `std::slice::from_ref(&x)` in `assumption_tree.rs` + `merkle.rs` test modules. Same allocator footprint, no clone, matches the lint's recommended idiom. - 2× `useless_vec`: `vec![0xE3, 0x00, 0x00]` → `[0xE3, 0x00, 0x00]` in serde-reject tests where the buffer is only borrowed. All 1011 unit tests pass. `cargo clippy --all-targets` is now clean. * ix check: print full content hashes + add --workers flag - The anon-mode progress / failure-log labels and the meta-mode hash-display fallback were truncating addresses to 16 hex chars (`@1f4b195aefa10e26`). Drop the `[..16]` slice so the full 64-char Blake3 hex is printed (`@1f4b195aefa10e2690d13c5b98b3d9124d3fbb5c…`), matching what tooling like `--fail-out` records and what the metadata-free workflow actually needs to identify a constant. - `--workers N` flag on `ix check` plumbs through the existing `IX_KERNEL_CHECK_WORKERS` env var that `resolve_kernel_check_workers` in `src/ffi/kernel.rs` reads. Useful for isolating per-worker memory cost (`--workers 1` lets you see the env's static footprint without per-worker overhead) and for capping concurrency in resource-tight contexts. * Mmap .ixe + cache-free LazyConstant for anon mode `lake exe ix check compilemathlib.ixe --anon` on a 3.2 GB env now peaks at ~11 GB RSS, down from ~40 GB; build_anon_work is 50× faster. Three intertwined changes: 1. Memory-map the .ixe (avoids ~3.2 GB heap copy of bytes) - Cargo: add memmap2 = "0.9". - `BytesSource` enum in `src/ix/ixon/lazy.rs` distinguishes heap-resident `Arc<[u8]>` from `(Arc<Mmap>, offset, len)` windows into a memory-mapped file. `LazyConstant::raw_bytes`, `verify_address`, `PartialEq` route through `BytesSource::as_slice` so every existing consumer is mmap-transparent. - `Env::get_anon_mmap(path)` in `src/ix/ixon/serialize.rs` opens the file, mmaps it, and stores Section-2 consts as mmap-backed `LazyConstant`s. Sections 3-4 (names + named) are still parsed transiently to harvest hints, then dropped before return. - `rs_kernel_check_anon` (`src/ffi/kernel.rs`) switches to `get_anon_mmap`; the old `std::fs::read` + `Env::get_anon` heap path is gone for the anon FFI. 2. Strip the persistent `LazyConstant` parsed-Constant cache - `LazyConstant.cache` was `Arc<OnceLock<Arc<Constant>>>` and accumulated forever — for mathlib that meant ~30 GB of parsed `Arc<Expr>` trees pinned in the env across the entire run. - New shape: `cache: Option<Arc<Constant>>`. Populated only by `from_constant` (compile-side, where we already own the parsed value). `from_bytes`/`from_mmap_slice` set `None`, and their `get()` parses fresh on every call without storing. - Re-parse cost is bounded by the existing per-worker dedup: `kenv.consts` already addresses each ingressed constant once per work item, and `clear_releasing_memory()` drops the kenv between items. The kenv is now the only persistent materialization layer. 3. `LazyConstant::peek_variant` for `build_anon_work` - One-byte read of the outer Tag4 head to identify the `ConstantInfo` variant — no body parse, no allocation. - `build_anon_work` now dispatches on `peek_variant()`; only `Muts` blocks trigger `lc.get()` (we need the member list for projection-address enumeration), and that `Arc<Constant>` drops at the end of the match arm. - Previously every constant was fully materialized at startup just to read its variant tag. With ~95% of the env being standalone/projection, the work-enumeration pass now skims the env at near-IO speed. Test updates: - `mmap_slice_roundtrips` (already added with the earlier mmap scaffolding) exercises the mmap window roundtrip. - New `from_bytes_does_not_cache` and `from_constant_clones_share_cache` document the new caching contract. - `peek_variant_*` tests cover every variant + empty-bytes and unknown-flag error paths. - `lazy_sparsity_only_materializes_closure` in `src/ix/ixon/env.rs` reframed to assert BFS-of-closure correctness instead of cache side-effects (`is_materialized()` no longer fires for lazy loads). End-to-end on compilemathlib.ixe (--anon, 32 workers): before: 640658/640658 in 266s, peak RSS ~40 GB after: 640658/640658 in 223s, peak RSS 11.3 GB * Address review findings: correctness + small refactors Code-review pass over the anon-mode / mmap / cache-strip work. This commit lands the highest-value subset — the correctness fixes and the small refactors that prevent future drift — and leaves the larger items (format-version bump, sealed marker trait, AnonEnv audit) for separate follow-ups. Correctness: - Verify per-constant address on load (#1). `LazyConstant::verify_address` existed but was never invoked from `Env::get`, `get_anon`, or `get_anon_mmap`. The env-level merkle root catches missing/extra entries but not byte-tampering of a constant whose key is intact; without this check, corruption surfaced much later inside `LazyConstant::get` with a misleading parse error. Inline the `Address::hash(bytes) == addr` check in each loader's Section-2 loop. Added 3 corruption-detection tests (`env_const_bytes_tampering_*`). This required updating a handful of existing tests that stored constants under fake `Address::hash(b"a")` keys instead of their true content hashes — round-tripping such envs now correctly rejects. Added a `store_canonical(env, c) -> Address` test helper for the canonical pattern, and `*_discriminator(refs, n)` so tests can produce content-distinct constants when the same ref-set would otherwise collide. - Hard-error in `ixon_env_to_decoded` on parse failure (#6). `src/ffi/ixon/env.rs` used `filter_map` to silently drop any const whose bytes failed `LazyConstant::get` — the Lean caller had no signal of the lost entries. Switch to a Result-collecting loop and propagate the first parse error; update both FFI call sites (`rs_de_env`, `rs_de_env_anon`). - Doc fixes (#4, #5). `docs/Ixon.md`'s AssumptionTree section described only `Leaf=0x00` and `Node=0x01`, missing `Padding=0x01` (and the docs' `Node` tag collided with the real `Padding` tag — the real `Node` is `0x02`). Also: the env-section table cell said "opt-tagged merkle root" but the implementation writes a bare 32-byte address. Refactors that prevent future drift: - Canonical projection-address helpers (#9). Four production sites reconstructed `Constant::new(IxonCI::{D,I,R,C}Prj{...}).commit().0` by hand; the anon pipeline silently breaks if any one drifts. Extract `defn_proj_address` / `indc_proj_address` / `recr_proj_address` / `ctor_proj_address` (plus `_constant` variants) in `src/ix/ixon/constant.rs`. Update `compile.rs`, `compile/mutual.rs`, and the anon `*_proj_addr` helpers to call them. - `verify_proj_addr_in_env` helper (#21). `ingress_anon_block` had the same "computed-address not in env" check repeated four times (DPrj/RPrj/IPrj/CPrj). DRY into one helper that produces a consistent error format. UX: - Anon display labels switched to `#<hex>` everywhere (#18). Rust was emitting `@<hex>` in progress/fail-out; Lean's `runCheckAnon` was emitting `#{i}` (result index). Standardize on `#<hex>` so the CLI failure summary is joinable with the fail-out file. This required exposing addresses per result slot to Lean — `rsCheckAnonFFI` now returns `Array (String × Option CheckError)` pairing each result with its content-address hex string. Rust builds the pairs via a new `build_anon_result_array` helper. - Stale "check-ixon" header in FailureLog (#16). One-line rename in the doc comment + `# ix check-ixon failures` writeln to `# ix check failures`. Tests: - `testPrimitivesParity` (PrimAddrs regen-parity test). Catches silent drift between hardcoded primitive addresses in `PrimAddrs::new()` and what `rsCompileEnvFFI` produces from the live Lean primitives. Plumbing: new `PrimAddrs::lean_parity_table()` returns `(lean_name, hex)` pairs; new `rs_prim_addrs_canonical` FFI exposes them to Lean; new test in `BuildPrimitives.lean` iterates `kernelPrimitives`, compares against the hardcoded table, and fails with a printable diff on mismatch (with instructions to regenerate). Skipped: `eagerReduce` — synthetic kernel marker whose PrimAddrs value (`0xff…3`) intentionally diverges from its compiled content hash (which collides with `id`). - 3 new corruption-detection tests in `serialize.rs` exercise the Section-2 verify check for `Env::get`, `Env::get_anon`, `Env::get_anon_mmap`. Verification: `cargo test --lib` 1025 passing (3 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe` 105487/105487 in ~21s, `--anon` 89010/89010 with peak RSS 1.4 GB, `.lake/build/bin/IxTests --ignored rust-kernel-build-primitives` shows both `build primitives dump` and `primitive address parity (PrimAddrs vs live compile)` pass. Out of scope (deferred to follow-ups): - #2 Format version bump (Env::FLAG) - #3 rs_kernel_check_consts_anon still uses Env::get on main - #8 Sealed marker trait for the lazy_anon transmute - #11/#12 AnonEnv audit (vestigial wrapper) - #13, #15, #17, #19, #20, #22-25 Various quality cleanups * Round-2 review fixes: mmap defense + small cleanup Six small items from the round-2 review of the anon-mode work: - N1: stat-at-open defense in `Env::get_anon_mmap` (`src/ix/ixon/serialize.rs`). Capture the file size before mmap; if the kernel's mapped length disagrees (file truncated between open and map) bail with a clear error instead of letting workers SIGBUS deep in `LazyConstant::get`. Truncate-in-place under a live mapping is still undefined per POSIX — documented as a caller contract — but the open-time check catches the common case. - New `get_anon_mmap_survives_file_unlink` test: load via mmap, unlink the path, then materialize both already-touched and not-yet-touched constants. Locks in the inode-retention invariant that the SIGBUS analysis depends on; a future refactor that switched to `mmap_anonymous` or copied bytes into a tmpfile would fail loudly here instead of letting workers SIGBUS in production. - #11: delete `AnonEnv::as_ixon_env_unchecked` (`src/ix/kernel/anon_env.rs`). Dead `pub(crate)` escape hatch marked `#[allow(dead_code)]` — confirmed zero callers and removed. - #13: `debug_assert!` on `OnceLock::set` for both result vectors (`src/ffi/kernel.rs`, meta + anon paths). The previous `let _ = results[idx].set(...)` silently dropped a re-set. If a future `build_*_work` dedup refactor breaks the one-write-per-slot invariant, debug builds now panic with the slot index instead of silently losing results. - #19: `allNames.contains` O(n²) preflight → `Std.HashSet` lookup (`Ix/Cli/CheckCmd.lean`). At mathlib scale (~700k env names × thousands of seed names) the previous linear scan-per-name spent measurable seconds on missing-name preflight alone. - N4: doc imprecision in `src/ix/ixon/lazy.rs` — the cache-policy preamble said the worker `KEnv` is "cleared between work items" but the actual cadence is `clear_every` items (`IX_KERNEL_CHECK_CLEAR_EVERY`, default 1 but tunable). Refined wording so the doc matches the code. Verification: `cargo test --lib` 1026 passing (1 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe --anon` 89010/89010 in 15.2s peak RSS 1.4 GB (regression-clean). Out of scope, separate follow-ups: - #2 Format version bump - #3 rs_kernel_check_consts_anon zkPCC FFI (Env::get → mmap) - #8 Sealed marker trait for the lazy_anon transmute - #10 child_arena closure side-effect (widen MField to tuple) - #12 AnonEnv duplicates bfs_refs / transitive_deps_excl - #14 anon worker bypasses M-generic display helpers - #17 Anon --fail-out docstring claims --consts-file compat - #20 Singleton-unwrap duplicated across compile paths - #22-24 Lean `partial def`, `.get!` in tests, unguarded as-u64 casts - N2/N3 Lean is_materialized() callers + compute_const_size_breakdown - Test gaps: rs_kernel_check_anon integration, concurrent mmap, verify_address-on-mmap-corruption, etc. * rustfmt * Tests: derive RawConst addrs from content (verify_address fix) The `Address::hash(bytes) == addr` defense added in f792102 rejects `RawConst { addr, const }` pairs where `addr` is uncorrelated with `serConstant const`. The Rust-side tests in `serialize.rs` were updated at the time (`store_canonical` helper), but three Lean-side sites still produced mismatched pairs and surface now as: × ∃: Env serialization Lean==Rust × ∃: serde RawEnv with data × ∃: serde RawEnv roundtrip deserialization failed: rs_de_env: Env::get: const at idx 0 bytes hash to 6110b739… but stored under b177ec1b… Three fixes, all derive `addr := Address.blake3 (serConstant c)`: - `Tests/Gen/Ixon.lean::genRawConst` — property-test generator was `RawConst.mk <$> genAddress <*> genConstant` (uncorrelated). Now generates the const first, then derives addr from `serConstant`. - `Tests/FFI/Lifecycle.lean::serdeTests` (`withData` unit case) — was using one `testAddr := Address.blake3 #[1,2,3]` for both the const and unrelated blob/comm slots. Split into `testAddr` (still used for the content-hash-free blob/comm) and `testConstAddr := Address.blake3 (serConstant testConst)`. Added a name entry for the new canonical addr. - `Tests/FFI/Lifecycle.lean::genSerdeRawEnv` — the "pool of addresses" pattern picked const addrs from a random pool that couldn't possibly match content hashes. Restructured: consts derive canonical addrs from content, each gets its own name-table entry appended to the pool-derived entries so the serde pipeline's "all addresses resolvable" invariant still holds. * Regenerate Aiur primitive addresses for singleton-unwrap Compiler's singleton non-inductive Muts unwrap (12630aa) changed the content hashes of 48 primitives (`Nat.rec`, `Nat.add`, `Nat.mul`, …). The Aiur kernel's `Ix/IxVM/Kernel/Primitive.lean` still held the pre-unwrap blake3 bytes, so primitive dispatch missed every affected constant and fell through to structural `Nat.rec` unfolding. Tests like `IxVMPrim.nat_mul_big` then drove millions of Nat.succ unfoldings inside the Aiur trace, OOMing the prover. Addresses regenerated from `PrimAddrs::lean_parity_table()` (which the branch had already updated in `src/ix/kernel/primitive.rs`). * Fix standalone Recr ingress: typ-based inductive lookup `find_matching_block_addr` heuristic in the standalone-Recr branch of `build_convert_inputs_walk` picks the inductive block by matching ctor count to rule count. When multiple in-scope inductives share the same ctor count, it returns the wrong block — the stored rules' `ctor_idx` then points to a sibling's ctor, while `populate_rules` (canonical) derives `ctor_idx` from the right inductive's `ctor_indices`. The mismatch surfaces as `compare_rules`'s `assert_eq!(s_ctor, c_ctor)` panic (e.g. `38 != 40` for `IxVMPrim.nat_land_lit`). Switch the standalone-Recr path to the same typ-based resolution `build_aux_recr_ctor_idxs` already uses for aux Recr blocks: peel `params + motives + minors + indices` foralls of `recr.typ`, take the major's head Ref, look it up in `refs` to get the inductive's address, then resolve `IPrj.block` for the Muts wrapper. Slice ctor positions for the specific member via `extract_member_ctor_idxs`. Also store the resolved Muts `block_addr` in `CKRecr` (was passing the heuristic's wrong address through to `convert_recursor`). --------- Co-authored-by: Arthur Paulino <arthurleonardo.ap@gmail.com> Co-authored-by: Samuel Burnham <45365069+samuelburnham@users.noreply.github.com>

plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed).

* Aiur Bytecode → Rust codegen for the IxVM kernel Adds a new Aiur pipeline stage that translates `Bytecode.Toplevel` into a Rust source module, one `fn aiur_fn_N` per Aiur function. The generated code mirrors `src/aiur/execute.rs`'s QueryRecord side effects exactly: same `function_queries.insert` timing, same cache-hit multiplicity bumps, same memory-queries insertion order, same `bytes{1,2}_queries` updates, same IO sequencing. Per-witness trace hashes match the interpreter byte-for-byte on every const tested (`Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). * `Ix/Aiur/Stages/Codegen.lean`: structured Rust IR (`RustExpr` / `RustStmt` / `RustItem` / `MatchArm` / labeled blocks) plus a formatter. Codegen is a structural walk over the Bytecode AST, no string templating. `EmitM = StateM EmitState` threads `(nextVal, nextLabel)` so per-ValIdx Rust locals and per-MC labels are fresh. * Aiur's `map: Vec<G>` is gone in the generated kernel: every ValIdx becomes a Rust local `__v_{i}: G`. Match-arm bodies snapshot `nextVal` on entry so per-arm allocations don't leak to siblings. `MatchContinue` / `Yield` use labeled-block + `break 'label [G; OUT_SIZE]` to bubble yielded values out and rebind them at outer scope. * Deep Aiur recursion uses `stacker::maybe_grow(64 KiB, 4 MiB, …)` at the top of every generated fn, so the native call stack grows on demand rather than pre-reserving a giant thread stack. Verified against shard 24 of the 64-way `init.ixes` partition, which SEGFAULTs without it. * `ix codegen`: new CLI command. Compiles the IxVM Aiur source, walks the bytecode, writes the generated Rust to a fixed path (`src/ix/aiur_ixvm.rs`). The output path is hard-coded; no override. * `src/ix/aiur_ixvm.rs`: the generated kernel, 743 `aiur_fn_*` + `execute_generated` dispatch. Auto-regenerated; do not edit. `#![cfg_attr(rustfmt, rustfmt::skip)]` + a broad `#![allow(unused_*, non_snake_case, clippy::all)]` since the layout is for the compiler, not humans. * `src/ix/aiur_ixvm_runner.rs`: `execute_ixvm(toplevel, fun_idx, args, io_buffer) -> Result<(QueryRecord, Vec<G>), ExecError>`. Same return shape as `Toplevel::execute`, but routes through `execute_generated`. * `src/aiur/synthesis.rs`: `AiurSystem::prove_ixvm(...)` — same shape as `prove`, but the execute step calls `execute_ixvm`. Verification-compatible (proofs from one path verify under the other). * `src/aiur/execute.rs`: helpers exposed for the codegen'd kernel (`bytes{1,2}_execute` → `pub(crate)`, `unconstrained_big_uint_div_mod_helper` extracted, `CodegenBytes{1,2}{Op,}` aliases re-exported, `QueryRecord::new` → `pub(crate)`, `ExecError::InvalidFunIdx` added). * `src/ffi/aiur/protocol.rs`: two new FFI exports — `rs_aiur_toplevel_execute_ixvm` and `rs_aiur_system_prove_ixvm`. Same wire format as the existing `_execute` / `prove` exports. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.executeIxVM` — mirror of `execute`. * `Ix/Aiur/Protocol.lean`: `AiurSystem.proveIxVM`. * `Ix/Cli/CheckCmd.lean`: `runCompiled` now dispatches through `executeIxVM`. The Rust bytecode interpreter is no longer reachable from `ix check` (the Lean-side `--interp` fallback stays for richer error diagnostics). * `Ix/Cli/ProveCmd.lean`: `proveOne` uses `proveIxVM`. * `Cargo.toml`: `stacker = "0.1"`. For witnesses where execute is the dominant work (single-const checks via `ix check 'X'`), the codegen'd kernel is ~1.6× faster than the bytecode interpreter end-to-end (measured on `Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). Tiny consts are within ~10% (stacker probe overhead amortises). Peak RSS is within a few MB of the interpreter — stacker grows the stack on demand, no giant upfront reservation. For shard-driven `ix check --shard K` runs the speedup is invisible: per-shard wall time is dominated (~92%) by `buildShardCheckEnvWitness` in Lean, NOT by the kernel execute (~8%). Cutting witness construction is the next lever. `Nat.add_comm` proof produced via `proveIxVM` verifies under the existing `AiurSystem::verify`. End-to-end: lake exe ix prove 'Nat.add_comm' → proof addr 0d0ab0f9… lake exe ix verify 0d0ab0f9… → ok * Move shard witness construction to Rust + parallelise Replaces `IxVM.ClaimHarness.buildShardCheckEnvWitness` (Lean side, ~92% of shard wall time on heavy partitions) with a Rust port that builds the `aiur::execute::IOBuffer` directly, without per-byte boxing into Lean `Aiur.G` values. The two hot phases run on rayon: * Closure walk: each owned addr's transitive `Constant.refs` + projection-block traversal runs on its own thread; results are deduped through a `dashmap::DashSet`. * Byte-to-G conversion: per-const `(key, data)` tuples are built in parallel chunks of 256; final IOBuffer assembly (channel arena append + key→`IOKeyInfo` map insert) runs serially because the arena index is monotonic. * `src/ix/aiur_ixvm_witness.rs` (new): `build_shard_check_env_witness` produces `(claim, claim_digest_input, io_buffer)` ready to feed to `execute_ixvm`. Mirrors the 6-channel layout documented in `Ix/IxVM/ClaimHarness.lean` (claim / asm tree / const bytes / Defn hint / blob discriminator / blob raw bytes). * `src/ffi/aiur/protocol.rs`: new FFI `rs_aiur_toplevel_shard_check_ixvm` bundles witness build + `execute_ixvm` into one cross-language trip. Returns the same `(output, ioBuffer, queryCounts)` shape as `rs_aiur_toplevel_execute_ixvm`, so the Lean shim stays drop-in compatible. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.shardCheckIxVM` wraps the FFI. * `Ix/Cli/CheckCmd.lean`: shard-mode `ix check` (`--ixe + --ixes`) now dispatches through `runShardOwnedNative` → `shardCheckIxVM`, bypassing the Lean witness builder. Single- shard and whole-partition paths share the fast route; the legacy `runShardCheckManifest` / `runShardCheckAll` callbacks stay live for `--interp` only. Shard 26 of the 64-way `init.ixes` partition, on the same host: | Variant | total | speedup | | --- | --: | --: | | Lean witness + bytecode-interp (baseline)| 1029 s | 1.0× | | Lean witness + codegen kernel | 1015 s | 1.01× | | Rust witness (serial) + codegen | 101 s | 9.95× | | Rust witness (parallel) + codegen | 80 s | 12.9× | The serial Rust witness collapses the 935 s of Lean per-byte boxing into ~23 s; rayon hides that 23 s under the codegen-kernel execute (~78 s) so witness build effectively becomes free at the shard level. FFT cost matches the bytecode interpreter exactly (107_006_963_281), so the QueryRecord layout is bit-identical across all four variants. * Closure walk uses single-source BFS with the global visited set shared via `DashSet::contains` checks before pushing to the per-thread stack — avoids redundant work without a per-owned union step. * Per-channel ordering: each chunk's partial `ChannelEntries` feeds the serial fold in iteration order, so channel arena contents match the Lean side's exactly. Test still relies on trace-hash parity from the earlier codegen commit, which exercised the same path through the bytecode interpreter and the codegen kernel. * Coverage check is currently skipped on the IxVM-native whole-partition path (`runShardManifestAllNative`); legacy `runShardCheckAll` still does it for `--interp`. Re-enable separately if needed. * Codegen: value-returning byte ops, dead-fold the unconstrained plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed). * Wire Rust witness fast path through every check/prove entry point Per-claim mode (a `Claim.check addr none`) builds a closure rooted at the target address. That closure can be the whole environment when the target is a heavily-shared root constant. The Lean witness builder paid per-byte boxing into `Aiur.G` for every byte in that closure — the same cost we already eliminated for shard mode. # Surface Five new FFIs, all bundling witness build + execute_ixvm (and STARK prove where applicable) into one cross-language trip so the `IOBuffer` never crosses the boundary mid-pipeline: * `rs_aiur_toplevel_check_addr_ixvm` — `Bytecode.Toplevel.checkAddrIxVM (toplevel) (funIdx) (ixePath) (addrBytes)`: per-claim check; builds witness for `Claim.check addr none` via `build_claim_check_witness`, runs `execute_ixvm`. Same return shape as `shardCheckIxVM`. * `rs_aiur_toplevel_check_env_bytes_ixvm` — same as above but takes the env as a serialized byte blob (`Ixon.serEnv`) instead of a `.ixe` path. Used by the compiled-Lean-env code path (`ix check NAME` without `--ixe`), where the env is built in Lean memory. * `rs_aiur_system_prove_addr_ixvm` — `AiurSystem.proveAddrIxVM (...)`: per-claim prove (witness + execute + STARK prove all in Rust). * `rs_aiur_system_prove_env_bytes_ixvm` — bytes-blob counterpart. * `rs_aiur_system_shard_prove_ixvm` — `AiurSystem.shardProveIxVM (...)`: per-shard prove end-to-end in Rust. `build_claim_check_witness` lives next to `build_shard_check_env_witness` in `src/ix/aiur_ixvm_witness.rs` and uses the same parallel closure walk + parallel byte→G conversion. The bytes-blob FFIs additionally harvest `anon_hints` from each `Def` named entry after `Env::get` — `Env::get` (full form, used by the bytes-blob path) doesn't populate `env.anon_hints` the way `Env::get_anon` does, but the kernel's `verify_claim` reads ch 3 (Defn reducibility hints) so the hints must end up in the env. This mirrors the harvest pass already inside `get_anon` at `src/ix/ixon/serialize.rs:1683`. # Wiring `Ix.Cli.CheckCmd.WitnessSource`: a three-arm discriminated union threaded through `forEachClaim` (the shared check/prove driver): * `.native ixePath addr` — `.ixe`-backed env, Rust mmap. * `.nativeBytes envBytes addr` — Lean-memory env serialized to bytes, Rust decodes via `Env::get`. * `.lean witness` — pre-built `ClaimWitness` (fallback). `runCompiled` and `proveOne` dispatch on the source. # Coverage | Command | Path | |---------------------------------------|-----------------------------------------------| | `ix check NAME --ixe` | **`checkAddrIxVM`** (new) | | `ix check NAME` (no `--ixe`) | **`checkEnvBytesIxVM`** (new) | | `ix check --claim hex --ixe` | `checkAddrIxVM` for `check addr none` | | `ix check --ixes --shard K` | `shardCheckIxVM` (existed) | | `ix check --ixes` | `shardCheckIxVM` (existed) | | `ix prove NAME --ixe` | **`proveAddrIxVM`** (new) | | `ix prove NAME` (no `--ixe`) | **`proveEnvBytesIxVM`** (new) | | `ix prove --ixes --shard K` | **`shardProveIxVM`** (new) | | `ix prove --ixes` | **`shardProveIxVM`** loop (new) | | `Benchmarks/Typecheck.lean` Phase 1 | `checkAddrIxVM` / `executeIxVM` | | `Benchmarks/Typecheck.lean` Phase 2 | `proveAddrIxVM` / `proveIxVM` | | `Benchmarks/IxVM.lean` | `proveIxVM` (was `prove`) | `--interp` route is preserved: it materialises any `WitnessSource` back into a `ClaimWitness` before driving the Aiur source interpreter. `--claim <hex>` over non-`check addr none` variants (`eval` / `reveal` / `contains` / `checkEnv-with-asm`) still uses the Lean witness builder. # Sanity * `ix check Nat.add_comm --ixe init.ixe` (warm): 3.1 s wall, FFT = 23_603_449. * `ix check Nat.add_comm` (compiled env via bytes blob, warm): 2.4 s wall, FFT = 23_603_449. # Known overheads (tracked for a follow-up redesign) * **Per-claim env re-parse on `--ixe + many names`**: each FFI call re-runs `Env::get_anon_mmap` on the same path. Mathlib-scale iteration pays the lazy-index build N times instead of once. * **`--interp + .nativeBytes` round-trip**: Lean serializes the compiled env then `materialise` deserializes it back, just to feed `runInterp`. * **`runShardProveNative` claim reconstruct**: redoes the closure walk + canonical `AssumptionTree` build Lean-side after `shardProveIxVM` already computed the same claim internally. A future `EnvHandle`-based API would close all three: build the env once into a Rust-owned handle, pass the handle to every per-claim/per-shard FFI, and have prove FFIs return the serialized claim bytes. * EnvHandle: Rust-owned env shared across per-claim / per-shard FFI calls Before: every per-claim and per-shard FFI re-parsed the Ixon env (`Env::get_anon_mmap` per call). On `--ixe + many names` / all-shards prove, this is the dominant overhead for iteration-heavy workflows — O(num_consts) lazy-index build × N targets. After: the env lives once per CLI invocation in a Rust-owned `EnvHandle`. Lean holds an opaque `Aiur.EnvHandle` reference and threads `@& EnvHandle` through every per-target FFI call. The env is parsed exactly once at handle construction; downstream calls share it. # Surface Five new FFIs collapse the previous six per-call variants (`checkAddrIxVM`, `checkEnvBytesIxVM`, `shardCheckIxVM`, `proveAddrIxVM`, `proveEnvBytesIxVM`, `shardProveIxVM` — all deleted): * `rs_aiur_env_handle_from_ixe(path) → LeanExternal<EnvHandle>`: mmap-load via `Env::get_anon_mmap`. Anon parser already harvests `anon_hints`; no post-pass. * `rs_aiur_env_handle_from_bytes(blob) → LeanExternal<EnvHandle>`: decode `Ixon.serEnv`-shape blob via `Env::get` + harvest `anon_hints` from each `Def` named entry. Used by the compiled-Lean-env path. * `rs_aiur_toplevel_check_addr_with_env(toplevel, fun_idx, handle, addr_bytes)`: per-claim check. Reuses handle's parsed env. * `rs_aiur_toplevel_shard_check_with_env(toplevel, fun_idx, handle, owned_blob)`: per-shard check. * `rs_aiur_system_prove_addr_with_env(system, fri, fun_idx, handle, addr_bytes) → (claim_bytes, proof, ioBuffer)`: per-claim prove. Rust serializes the reconstructed `Ix.Claim` via `ixon::Claim::put` so Lean can deserialize via `Ixon.runGet Ix.Claim.get` directly — no closure walk + canonical `AssumptionTree` recomputation Lean-side. * `rs_aiur_system_shard_prove_with_env(system, fri, fun_idx, handle, owned_blob) → (claim_bytes, proof, ioBuffer)`: per-shard prove. # Lean * `Aiur.EnvHandle` opaque type with `fromIxe` / `fromBytes`. * `Aiur.Bytecode.Toplevel.checkAddrWithEnv` / `shardCheckWithEnv`, `Aiur.AiurSystem.proveAddrWithEnv` / `shardProveWithEnv`. * `Ix.Cli.CheckCmd.Target` replaces `WitnessSource`: ```lean inductive Target where | addr (a : Address) -- Claim.check addr none | shard (owned : Array Address) -- Claim.checkEnv | leanW (w : ClaimWitness) -- --interp, --claim non-check ``` `runCompiled` / `proveOne` take `(envHandle?, target)`. The envHandle is `none` only for `.leanW` (`--interp` legacy path). * `runShardOwnedNative` and the manifest-driver helpers take the envHandle as a parameter. Single-shard mode builds it once; all-shards mode reuses the same handle across every shard's FFI call (eliminates per-shard re-mmap). * `runShardProveNative` deserializes the wire claim bytes via `Ixon.runGet Ix.Claim.get` instead of re-running `shardCheckEnvClaim`. # Benchmarks `Benchmarks/Typecheck.lean` builds the envHandle once before Phase 1, reuses it across Phase 1 (execute) + Phase 2 (prove). Both phases now go through `checkAddrWithEnv` / `proveAddrWithEnv` on the full-closure path. `--subject-only` still uses Lean `buildVerifyConst` + `executeIxVM` / `proveIxVM` (witnesses intentionally small there). # New crate `crates/ix/src/env_handle.rs` — the `EnvHandle` struct + constructors live in the existing `ix` crate next to the witness builder and codegen runner. # Sanity * Multi-target check (warm): `ix check --ixe init.ixe Nat.add_comm Nat.add` → 3.3 s wall, single envHandle shared across both targets. * Shard 26 check (warm): `ix check --ixe init.ixe --ixes init.ixes --shard 26` → ~78 s wall, FFT = 107_006_963_281 (parity preserved). # Coverage | Command | Path | |--------------------------------------|-----------------------------------| | `ix check NAME --ixe` | `checkAddrWithEnv` | | `ix check NAME` (no `--ixe`) | `checkAddrWithEnv` + fromBytes | | `ix check --claim hex --ixe` | `checkAddrWithEnv` for check-none | | `ix check --ixes --shard K` | `shardCheckWithEnv` | | `ix check --ixes` | `shardCheckWithEnv` × all shards | | `ix prove NAME --ixe` | `proveAddrWithEnv` | | `ix prove --ixes --shard K` | `shardProveWithEnv` | | `ix prove --ixes` | `shardProveWithEnv` × all shards | | `Benchmarks/Typecheck` Phase 1+2 | `checkAddrWithEnv` / `proveAddrWithEnv` | Only `--interp` and `--claim hex` over a non-`check addr none` persisted claim still build a Lean `ClaimWitness`; both go via the `.leanW` target arm. * ix codegen: add --check flag for CI; gate stale generated kernel `ix codegen --check` compares the emitted Rust source against the on-disk `crates/ix/src/aiur_ixvm.rs` and exits 0 if identical, 1 otherwise. No write side effect. ~2 s warm; fast enough to gate on every PR. Wired into the `lean-test` CI job so a forgotten regen on a kernel-touching PR fails CI instead of merging stale generated code that drifts from the Bytecode → Rust emitter. * ix check: --interp {source|bytecode} to bypass codegen kernel Two interpreter modes behind a single `--interp` flag: * `--interp source`: Aiur source interpreter (`Aiur.runFunction` over the source-level `Decls`). Richer per-step error diagnostics. * `--interp bytecode`: generic Aiur bytecode interpreter (`Bytecode.Toplevel.execute`, `rs_aiur_toplevel_execute` route). Skips the `ix codegen` + `cargo build --release` cycle needed after editing `Ix/IxVM/*.lean` — the bytecode is rebuilt Lean-side at exe load. Slower per-check than the codegen kernel; ideal for tight iteration on the IxVM source. * omit the flag entirely for the native codegen kernel (default). Invalid values (e.g. `--interp foo`) fail fast with a clear error. # Plumbing The `check_addr_with_env` and `shard_check_with_env` FFIs take a `use_bytecode: bool` and dispatch via a shared `dispatch_execute` helper. `--interp bytecode` sets it to `true`. The `.leanW` arm of `runCompiled` also picks between `Bytecode.Toplevel.execute` and `executeIxVM` based on the same flag. The `--interp source` path requires a Lean `ClaimWitness`. The existing driver had switched to `.addr`/`.shard` targets against the Rust-owned `EnvHandle`; source-interp had no way to materialise those. `forEachClaim` now takes a `forceLeanWitness : Bool` — when true, it builds a Lean witness for every target (via `mkWitness` / `loadIxonEnv` for the compiled-Lean-env path) and passes `.leanW` so `runInterp` can consume it directly. # Sanity * `ix check Std.Time.Week.Offset.ofMilliseconds` (warm): 9.3 s wall, FFT = 12_430_516_949 (codegen kernel). * `ix check --interp bytecode Std.Time.Week.Offset.ofMilliseconds` (warm): 14.6 s wall, FFT = 12_430_516_949 (bytecode interp). * `ix check --interp source Eq` (warm): 6.3 s wall, output `Eq: ()`. * `ix check --interp garbage Nat.add_comm`: exits 1 with clear error. * IxVM codegen: regen + rebase-fixups after KLevel port Aligns the ap/codegen-ixvm-native stack with main's KLevel pointer port (b84a500) and re-lands clippy-clean under the workspace lint set: - Regen crates/ix/src/aiur_ixvm.rs from Aiur source (KLevel = &KLevelNode changes the generated fn shapes; ~26k lines net churn). - Extend the generated file's #![allow(...)] header (via Ix/Aiur/Stages/Codegen.lean) with clippy::ptr_as_ptr, clippy::match_same_arms, and clippy::large_types_passed_by_value — three lints the codegen's straight-line style trips deterministically on every regen and that clippy::all doesn't cover. - rustfmt reflow in crates/aiur/src/execute.rs and a handful of ix / ffi files that CI now enforces via the ap/aiur-aux-recursor-parity toolchain. - Fix 4 real clippy warnings uncovered under the workspace lint set: * env_handle.rs: clone-on-Copy → deref (ReducibilityHints is Copy). * aiur_ixvm_witness.rs: collapse if-let-if, drop needless continue, slice::from_ref instead of &[x.clone()]. * aiur_ixvm_runner.rs: keep execute_ixvm's Vec<G> arg (required by AiurSystem::prove_ixvm's fn-pointer bound) + local #[allow( clippy::needless_pass_by_value)] with a comment explaining why. * ffi/aiur/protocol.rs: unqualify aiur::G/IOBuffer/QueryRecord after the workspace lint added unused_qualifications; is_multiple_of() over % 0. Verified: cargo clippy --workspace --all-targets --all-features --D warnings clean; lake test -- --ignored ixvm passes. * ix-check + ixvm tests: coverage gate, --interp source fix, codegen parity + rename ix crate to ixvm-codegen Four review fixes on the codegen-ixvm-native stack. - **Coverage gate on \`--ixes\` (all-shards, native path).** \`runShardManifestAllNative\` now calls \`shardsCover\` before running any shard, matching the pre-refactor \`runShardCheckAll\` behavior. Without this, a stale/truncated \`.ixes\` manifest makes \`ix check --ixes\` exit 0 while some env constants are never checked by any shard — the mundane trigger being: edit a definition, rebuild the \`.ixe\`, forget to re-run \`ix shard\`, and the constants that go unchecked are exactly the ones just edited. Single-shard \`--shard K\` path is unchanged (consistent with pre-refactor). \`shardsCover\` moved above \`runShardManifestAllNative\` to satisfy forward-decl ordering. - **\`ix check --ixe … --claim <hex> --interp source\` regression.** The \`claimHex\` arm in \`forEachClaim\` ignored \`forceLeanWitness\` and always mapped \`.check addr none\` claims to \`Target.addr\`. Under \`--interp source\`, \`runInterp\` then rejects \`.addr\` with "\`--interp requires a Lean witness; addr/shard targets unreachable here\`". Now mirrors the sibling name-based arms and routes through the Lean witness path when \`forceLeanWitness\` is on. Only bit the common claim shape. - **Codegen ↔ bytecode parity CI test.** New \`runParityCase\` + \`parityCases\` in \`Tests/Ix/IxVM.lean\` wire every constant already listed in \`kernelCheckEntries\` plus \`kernel_unit_tests\` through BOTH \`Toplevel.execute\` (bytecode) and \`Toplevel.executeIxVM\` (codegen'd Rust), and diff the returned \`(output, IOBuffer, QueryCounts)\` triples. Turns "generated kernel ≡ interpreter on QueryRecord" from reviewed-by-hand into checked-by-CI. 129 assertions run per suite invocation (43 constants × output + IOBuffer + QueryCount), all pass on this branch head. - **Rename \`crates/ix\` → \`crates/ixvm-codegen\`.** The old \`ix\` crate held only the codegen'd IxVM kernel + its runner + witness helpers. \`ix-kernel\` is the hand-written Rust ix typechecker; \`ix-common\` / \`ix-compile\` are Ix-level infrastructure. \`ixvm-codegen\` cleanly names what this crate holds. Path updates: workspace \`members\` and \`workspace.dependencies\`, ffi crate dep + \`use\` sites, Lean \`codegenOutPath\` in \`Ix/Cli/CodegenCmd.lean\`, doc reference in \`Ix/Aiur/Semantics/BytecodeFfi.lean\`. \`ix codegen\` now writes to \`crates/ixvm-codegen/src/aiur_ixvm.rs\`. Verified: \`cargo check --workspace\` + \`lake test -- --ignored ixvm\` both exit 0 post-rename; parity + pinned FFT tests all pass. * cargo-deny: allow ar_archive_writer's Apache-2.0 WITH LLVM-exception Post-rename, ixvm-codegen pulls stacker → psm → ar_archive_writer (a build-time transitive). ar_archive_writer is licensed under the LLVM-exception variant of Apache-2.0 (standard for LLVM tooling; no additional runtime obligation for downstream users), which isn't on the workspace allow list. Add a per-crate exception scoped to ar_archive_writer so we don't blanket-allow the license family across the tree.

arthurpaulino · 2025-02-04T13:52:08Z

No description provided.

Lays the groundwork for measuring the kernel performance plan (plans/okay-let-s-write-a-lucky-dolphin.md) per audit §10. Counters live in a new kernel::perf module; KEnv carries a PerfCounters field that is dumped from a Drop impl when IX_PERF_COUNTERS=1 is set. Unset is the production default — every increment short-circuits via a LazyLock<bool> so the cost is a single cached branch on the hot path. Wired into the cache get sites the audit identified: - whnf_cache hits/misses (whnf.rs around 201/211) - whnf_no_delta_cache hits/misses (whnf.rs around 437/446) - infer_cache and infer_only_cache hits/misses (infer.rs around 45/51) - def_eq_cache hits/misses (def_eq.rs around 137/149) - def_eq_failure set hits/inserts (def_eq.rs around 360) - per-constant peak/avg MAX_REC_FUEL consumption (recorded in TypeChecker::reset before the next constant resets it) Verified against lake test -- kernel-check-env --ignored: 192156/192156 constants pass in 251s (4% over the 242s baseline; overhead is the LazyLock branch + atomic load in the disabled path). P1: def_rank_id replaces def_weight_id for hint-priority comparison Audit Tier 1 #3 (kernel-perf-adversarial-audit-2026-04-26.md, §4.2): the prior u32 encoding mapped Abbrev to u32::MAX-1 and saturating-added Regular(h) to h+1, which collide at h ≥ u32::MAX-2. When that happens the delta-direction logic treats Abbrev and a maximally-heavy Regular as "same height" and unfolds both, instead of preferring Abbrev as Lean does (compare(d_t->get_hints(), d_s->get_hints()) at type_checker.cpp:910). Replace the u32 weight with a (class: u8, height: u32) tuple compared lexicographically: - Opaque / Theorem / unknown → (0, 0) - Regular(h) → (1, h) (height ordering preserved within class) - Abbrev → (2, 0) (strictly above every Regular) Update the two call sites (is_def_eq lazy-delta height comparison and lazy_delta_step). The map_or default for "missing head" is preserved as (u8::MAX, u32::MAX) — the branch is dead in practice (a_delta && b_delta imply both heads are present) but kept consistent with the prior u32::MAX sentinel. Two regression tests: - def_rank_abbrev_above_saturated_regular: Abbrev outranks Regular(u32::MAX) (the previous saturation collision). - def_rank_regular_orders_by_height: height monotonically orders Regular ranks within the class. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 256s (no regression vs the 242s pre-perf-counters baseline; the +14s is from the IX_PERF_COUNTERS=unset LazyLock branch added in the prior commit, not this change). P2: peel_proj_forall fast-paths syntactic Pi in projection inference Audit Tier 1 #2 (kernel-perf-adversarial-audit-2026-04-26.md, §7.2): infer_proj's two parameter-consuming loops (param peel and field peel) called self.whnf(&r)? unconditionally per iteration, on a body mutated by subst at the previous step. The whnf cache rarely hits between iterations and each call re-traverses the substituted body. Extract a peel_proj_forall(&r, err) helper that: - tries ExprData::All(..) syntactically first (no WHNF call), and - falls back to full self.whnf(e) only when the binder isn't already syntactic Pi. This mirrors Lean's inferProj at type_checker.cpp:582–610. Both projection-inference loops now call peel_proj_forall instead of unconditional whnf. Behaviorally equivalent — same WHNF semantics on miss, no semantic change otherwise. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 258s (parity with the post-pre-work, post-P1 baseline; no measurable regression and the cache-hit-rate counters will move on tactic-heavy workloads under IX_PERF_COUNTERS=1). P3a: WhnfFlags substrate (no behavior change) Lays the foundation for the Lean4Lean architectural alignment described in plans/okay-let-s-write-a-lucky-dolphin.md. Phase 3a is substrate-only — no call site is migrated to cheap mode yet, so behavior is unchanged. Adds: - WhnfFlags { cheap_rec, cheap_proj } with FULL and CHEAP consts and is_full(). CHEAP is currently equal to FULL until Phase 3c wires it. - whnf_core_with_flags (private): the existing whnf_core impl, now threading flags into recursive calls and try_iota_with_flags. - whnf_core / whnf_core_cheap (super): FULL/CHEAP wrappers. - whnf_no_delta_with_flags (private): the existing whnf_no_delta impl with the Prj branch gated on cheap_proj — falls back to full whnf on the projected value when not cheap. - whnf_no_delta (pub) / whnf_no_delta_cheap (super): wrappers. - try_iota_with_flags: gates major-premise WHNF and string-literal constructor reduction on cheap_rec. - try_proj_app_reduce_with_flags: gates projected-value WHNF on cheap_proj. Cache reads/writes (whnf_no_delta_cache, equiv-manager second-chance) are gated on flags.is_full(): cheap callers neither read nor write the cache, preserving the invariant that any cached entry is a fully-reduced normal form. Phase 3b will inline the projection branch into whnf_core to match Lean4Lean's two-layer architecture (refs/lean4lean/Lean4Lean/ TypeChecker.lean:266, 297). Phase 3c will flip CHEAP to enable cheap_proj and migrate specific def-eq sites. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 243s (matches the 242s baseline; substrate adds no measurable overhead when CHEAP == FULL). P3b: inline projection into whnf_core (Lean4Lean architectural alignment) Move the Prj branch from whnf_no_delta_with_flags into whnf_core_with_flags so our whnf_core matches Lean4Lean's whnfCore semantics exactly (refs/lean4lean/Lean4Lean/TypeChecker.lean:284-292, 337-341). Before this commit: whnf_core — beta + zeta + iota + cheap projection (recursive whnf_core on val, no delta) whnf_no_delta — whnf_core + FULL projection (full whnf on val) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta Lean4Lean's architecture has no whnf_no_delta layer. Their whnfCore includes projection, with the cheap_proj flag deciding whether the projected value uses whnfCore (cheap) or whnf (full). After this commit: whnf_core (with WhnfFlags) — beta + zeta + iota + projection (cheap_proj controls val reduction) whnf_no_delta — whnf_core(_, FULL) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta The bare-Prj branch in whnf_no_delta_with_flags is removed — whnf_core now handles it directly. The App-of-Prj branch stays in whnf_no_delta because whnf_core's loop returns once the outermost Prj is resolved; try_proj_app_reduce_with_flags gives one more attempt at the same cheap_proj policy when the outer expression is App(Prj, ...). Pure refactor, no semantic change with CHEAP == FULL. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 270s (within noise of the 243s pre-refactor baseline). Phase 3c will flip CHEAP to enable cheap_proj=true and migrate def-eq's lazy-delta sites surgically per Lean4Lean's pattern. P3c (postponed): document the HeaderParsedSnapshot regression P3a (substrate) and P3b (Lean4Lean architectural alignment) are committed. Phase 3c — flipping CHEAP to enable cheap_proj=true and migrating the def-eq lazy-delta sites — was attempted but reproduced 5 failures on chained projections in Lean.Language.Lean.HeaderParsedSnapshot.* even after P3b inlined the projection branch into whnf_core. The substrate is left in place; CHEAP stays equal to FULL until the regression's root cause is understood. Notes on the regression for the next investigator: - Failures: HeaderParsedSnapshot.{stx,result?,metaSnap,toSnapshot,ictx}, all with 'projection: type mismatch with declared struct'. - The struct `extends` a parent, so each projection is a chained Prj whose val is itself a Prj into the parent. - The error comes from infer.rs's infer_proj at the head-vs-struct_id address compare, after FULL whnf on val_ty. That whnf is FULL, but val_ty was inferred via paths that may have consulted a def-eq cache populated under cheap mode. Possible cache-poisoning suspect: def_eq_cache writes a `false` result for inputs whose lazy-delta loop bottomed out under cheap projections. The cache key uses raw a/b hashes, not cheap-reduced shapes, so a stored `false` is indistinguishable from a FULL `false` by future readers. - Lean4Lean does not have an analogous wide def_eq_cache; their failure cache is keyed only on same-spine pairs in lazyDeltaReductionStep. Future P3c iterations should either prove the cache poisoning theory incorrect or restrict def_eq_cache writes to FULL-derived results.

* Refactor Claim ADT + add merkle infrastructure for ZK aggregation Adds the format hooks needed before recursive verification lands in Aiur. Five-variant Claim ADT with explicit assumption commitments, a canonical Blake3 merkle module, a serializable AssumptionTree for recovering leaf sets, and a Contains claim for the discharge step. Claim ADT (5 variants): - Eval { input, output, assumptions: Option<Address> } - Check { const_addr, assumptions: Option<Address> } - CheckEnv { root, assumptions: Option<Address> } - Reveal { comm, info } -- unchanged, no assumptions - Contains { tree, const_addr } -- new, for inclusion proofs Tag4 reorganized to keep everything in single-byte tags: - 0xE for env, comm, AssumptionTree, and claims (slots 0-7) - 0xF for proofs (slots 0-4; 5-7 reserved) - Comm moved from variant 5 -> 1 Matches the "Variant (0-7)" constraint documented in docs/Ixon.md. Env serialization: - Every .ixe file now carries a canonical merkle root over its consts.keys() in the on-disk header (non-optional, 32 bytes; empty const sets use the zero-address sentinel). - Two envs with the same const set produce byte-identical roots regardless of insertion order. Verified on deserialize. New modules: - src/ix/ixon/merkle.rs + Ix/Merkle.lean: canonical sorted builder, free-form merkle_join composition, membership proofs, domain separation per RFC 6962. - src/ix/ixon/assumption_tree.rs + Ix/AssumptionTree.lean: serializable merkle tree with Leaf/Padding/Node variants. canonical() builds the same shape merkle_root_canonical hashes; join() is O(1) free-form composition. - src/ix/kernel/claim.rs: builders that compute transitive-dep assumptions from an env (build_check_claim, build_eval_claim, build_check_env_claim, env_merkle_root). Other: - Extracted shared BFS walker on Env::bfs_refs + Env::transitive_deps_excl; the inlined test-feature copy in lean_env.rs now calls into it. - Lean FFI export rs_env_merkle_root for cross-impl verification of the env root. - Proof bytes for all variants are uniform opaque ZK bytes; witness data (e.g., Contains merkle paths) is prover-side scratch consumed by the ZK circuit and not transmitted on the wire. - docs/Ixon.md Tag4 tables and env section updated; .ixe extension documented. Recursive verification (the ZK proof generation for Contains and the aggregation discharge transitions) is intentionally deferred to a follow-up. Tests: 993 Rust unit tests pass (was 953 pre-refactor), 813 Lean tests pass with no failures; cargo clippy clean. * Add lazy env deserialization and anon-mode kernel FFI Two related changes to reduce kernel memory + lay groundwork for metadata-isolated typechecking. **Lazy constant deserialization.** `Env::consts` now stores `LazyConstant` (`Arc<[u8]>` + `OnceLock<Arc<Constant>>`) instead of `Constant`. Constants are materialized on first access and the structured form is cached. Sparse access patterns (single-constant typecheck, transitive_deps walk, claim builders) only parse the closure they need; non-reachable constants stay as raw bytes. The `.ixe` section-2 layout gains a Tag0 length sidecar before each constant's Tag4 bytes: old: [addr:32] [Tag4 constant bytes] new: [addr:32] [Tag0 length] [Tag4 constant bytes] The length is section-level framing, not part of the constant's content hash. `Address::hash(raw_bytes) == addr` is preserved. `Constant::put`/`Constant::get` are unchanged. The Lean side (`Ix.Ixon.putEnv`/`getEnv`) reads/writes the new format. **Anon-mode kernel FFI.** New `rs_kernel_check_consts_anon(path, addrs, quiet)` exposes anonymous-mode typechecking by content address. The kernel runs as `KEnv<Anon>` / `TypeChecker<Anon>` with every `M::MField<T>` erased to `()`, so the typechecking logic structurally cannot read metadata. Useful for zkPCC verifiers that hold only addresses. Supporting pieces: - `KernelMode::HAS_META: bool` for future compile-time gating. - `AnonEnv<'a>` wrapper (`src/ix/kernel/anon_env.rs`) exposing only consts/blobs/transitive walks — no `named`/`names`/`comms`. - `Env::get_anon` reads header + blobs + consts, parse-and-drops metadata sections (3-5), returns an `Env` with empty `named`/ `names`/`comms`. Same merkle-root verification as `Env::get`. - `rs_de_env_anon` FFI + `Ix.Ixon.rsDeEnvAnon` Lean wrapper. - `Ix.KernelCheck.rsCheckConstsAnonFFI` Lean binding. Caveat: `ixon_ingress::<Anon>` still consults `Env::named` internally to enumerate work items. The resulting `KEnv<Anon>` is metadata-free so the typechecker is anon, but full ingress-level metadata isolation is a follow-up. Tests: 16 new (9 lazy + sparsity; 3 AnonEnv; 4 get_anon). All 1009 Rust unit tests pass; all 813 Lean tests pass; clippy clean. * Simplify ix check + add metadata-free anon mode - `ix check` becomes .ixe-only with a positional `<path>`. Drops the `--lean` compile-and-check flow and the `--env` flag; direct Lean → kernel checks remain available through `rsCheckConstsFFI` for tests. Removes the redundant `CheckIxonCmd.lean`. - New `--anon` flag runs a metadata-free kernel check: loads the .ixe via `Env::get_anon` (discards named/names/comms), enumerates work items from `env.consts` alone, and rebuilds member + ctor projection addresses deterministically via `Constant::commit`. Rejects `--consts`/`--ns`/`--consts-file` since it always checks everything. - Compiler: unwrap singleton non-inductive Muts blocks to standalone Defn/Recr. `Expr::Rec(0, univs)` already resolves correctly against a one-member block, so the wrapper was pure overhead; this keeps the env structurally uniform and matches `compile_single_def`. The anon ingress for standalones passes `mut_ctx_override = [self_id]` so self-recursive standalones still typecheck. - Kernel: anon parallel runner mirrors `run_checks_parallel_on_large _stacks` (32 workers, slow-detection, RSS tracking). Genericized `check_one_const<M>` / `check_consts_loop<M>` / `format_tc_error<M>` to share the runner. `KernelMode::meta_field_with` / `meta_field_try` gate metadata lookups in expression ingress at compile time. - Anon lazy ingress (`LazyAnonIngress` in tc.rs, helpers in ingress.rs) dedupes blocks via `kenv.blocks.contains_key(&KId<Anon>(B, ()))` before re-running `ingress_anon_block` (prevents N² re-ingestion when sibling projections fault separately within one worker check). - `Env::get_anon` now harvests `ReducibilityHints` from the otherwise-discarded Named section into a sidecar `Env::anon_hints: FxHashMap<Address, ReducibilityHints>`. Anon ingress threads these through a new `hints_override` parameter on `ingress_defn` so the lazy-delta tiebreak in `def_eq::def_rank_id` sees realistic heights instead of `Regular(0)`. Hints are performance advice, not correctness data — supplying them in anon mode preserves the metadata-free trust model. - Regenerate the canonical addresses in `PrimAddrs::new()`; singleton- unwrap changed the content hash of every self-recursive primitive (Nat.add, String, Nat.rec, …). End-to-end on compileinitstd.ixe (105,487 constants): meta: 105487/105487 in 20.6s, peak RSS 7.4GB anon: 89010/89010 in 17.8s, peak RSS 4.0GB * Clean up cargo clippy --all-targets - 11× `cloned_ref_to_slice_refs`: `&[x.clone()]` → `std::slice::from_ref(&x)` in `assumption_tree.rs` + `merkle.rs` test modules. Same allocator footprint, no clone, matches the lint's recommended idiom. - 2× `useless_vec`: `vec![0xE3, 0x00, 0x00]` → `[0xE3, 0x00, 0x00]` in serde-reject tests where the buffer is only borrowed. All 1011 unit tests pass. `cargo clippy --all-targets` is now clean. * ix check: print full content hashes + add --workers flag - The anon-mode progress / failure-log labels and the meta-mode hash-display fallback were truncating addresses to 16 hex chars (`@1f4b195aefa10e26`). Drop the `[..16]` slice so the full 64-char Blake3 hex is printed (`@1f4b195aefa10e2690d13c5b98b3d9124d3fbb5c…`), matching what tooling like `--fail-out` records and what the metadata-free workflow actually needs to identify a constant. - `--workers N` flag on `ix check` plumbs through the existing `IX_KERNEL_CHECK_WORKERS` env var that `resolve_kernel_check_workers` in `src/ffi/kernel.rs` reads. Useful for isolating per-worker memory cost (`--workers 1` lets you see the env's static footprint without per-worker overhead) and for capping concurrency in resource-tight contexts. * Mmap .ixe + cache-free LazyConstant for anon mode `lake exe ix check compilemathlib.ixe --anon` on a 3.2 GB env now peaks at ~11 GB RSS, down from ~40 GB; build_anon_work is 50× faster. Three intertwined changes: 1. Memory-map the .ixe (avoids ~3.2 GB heap copy of bytes) - Cargo: add memmap2 = "0.9". - `BytesSource` enum in `src/ix/ixon/lazy.rs` distinguishes heap-resident `Arc<[u8]>` from `(Arc<Mmap>, offset, len)` windows into a memory-mapped file. `LazyConstant::raw_bytes`, `verify_address`, `PartialEq` route through `BytesSource::as_slice` so every existing consumer is mmap-transparent. - `Env::get_anon_mmap(path)` in `src/ix/ixon/serialize.rs` opens the file, mmaps it, and stores Section-2 consts as mmap-backed `LazyConstant`s. Sections 3-4 (names + named) are still parsed transiently to harvest hints, then dropped before return. - `rs_kernel_check_anon` (`src/ffi/kernel.rs`) switches to `get_anon_mmap`; the old `std::fs::read` + `Env::get_anon` heap path is gone for the anon FFI. 2. Strip the persistent `LazyConstant` parsed-Constant cache - `LazyConstant.cache` was `Arc<OnceLock<Arc<Constant>>>` and accumulated forever — for mathlib that meant ~30 GB of parsed `Arc<Expr>` trees pinned in the env across the entire run. - New shape: `cache: Option<Arc<Constant>>`. Populated only by `from_constant` (compile-side, where we already own the parsed value). `from_bytes`/`from_mmap_slice` set `None`, and their `get()` parses fresh on every call without storing. - Re-parse cost is bounded by the existing per-worker dedup: `kenv.consts` already addresses each ingressed constant once per work item, and `clear_releasing_memory()` drops the kenv between items. The kenv is now the only persistent materialization layer. 3. `LazyConstant::peek_variant` for `build_anon_work` - One-byte read of the outer Tag4 head to identify the `ConstantInfo` variant — no body parse, no allocation. - `build_anon_work` now dispatches on `peek_variant()`; only `Muts` blocks trigger `lc.get()` (we need the member list for projection-address enumeration), and that `Arc<Constant>` drops at the end of the match arm. - Previously every constant was fully materialized at startup just to read its variant tag. With ~95% of the env being standalone/projection, the work-enumeration pass now skims the env at near-IO speed. Test updates: - `mmap_slice_roundtrips` (already added with the earlier mmap scaffolding) exercises the mmap window roundtrip. - New `from_bytes_does_not_cache` and `from_constant_clones_share_cache` document the new caching contract. - `peek_variant_*` tests cover every variant + empty-bytes and unknown-flag error paths. - `lazy_sparsity_only_materializes_closure` in `src/ix/ixon/env.rs` reframed to assert BFS-of-closure correctness instead of cache side-effects (`is_materialized()` no longer fires for lazy loads). End-to-end on compilemathlib.ixe (--anon, 32 workers): before: 640658/640658 in 266s, peak RSS ~40 GB after: 640658/640658 in 223s, peak RSS 11.3 GB * Address review findings: correctness + small refactors Code-review pass over the anon-mode / mmap / cache-strip work. This commit lands the highest-value subset — the correctness fixes and the small refactors that prevent future drift — and leaves the larger items (format-version bump, sealed marker trait, AnonEnv audit) for separate follow-ups. Correctness: - Verify per-constant address on load (#1). `LazyConstant::verify_address` existed but was never invoked from `Env::get`, `get_anon`, or `get_anon_mmap`. The env-level merkle root catches missing/extra entries but not byte-tampering of a constant whose key is intact; without this check, corruption surfaced much later inside `LazyConstant::get` with a misleading parse error. Inline the `Address::hash(bytes) == addr` check in each loader's Section-2 loop. Added 3 corruption-detection tests (`env_const_bytes_tampering_*`). This required updating a handful of existing tests that stored constants under fake `Address::hash(b"a")` keys instead of their true content hashes — round-tripping such envs now correctly rejects. Added a `store_canonical(env, c) -> Address` test helper for the canonical pattern, and `*_discriminator(refs, n)` so tests can produce content-distinct constants when the same ref-set would otherwise collide. - Hard-error in `ixon_env_to_decoded` on parse failure (#6). `src/ffi/ixon/env.rs` used `filter_map` to silently drop any const whose bytes failed `LazyConstant::get` — the Lean caller had no signal of the lost entries. Switch to a Result-collecting loop and propagate the first parse error; update both FFI call sites (`rs_de_env`, `rs_de_env_anon`). - Doc fixes (#4, #5). `docs/Ixon.md`'s AssumptionTree section described only `Leaf=0x00` and `Node=0x01`, missing `Padding=0x01` (and the docs' `Node` tag collided with the real `Padding` tag — the real `Node` is `0x02`). Also: the env-section table cell said "opt-tagged merkle root" but the implementation writes a bare 32-byte address. Refactors that prevent future drift: - Canonical projection-address helpers (#9). Four production sites reconstructed `Constant::new(IxonCI::{D,I,R,C}Prj{...}).commit().0` by hand; the anon pipeline silently breaks if any one drifts. Extract `defn_proj_address` / `indc_proj_address` / `recr_proj_address` / `ctor_proj_address` (plus `_constant` variants) in `src/ix/ixon/constant.rs`. Update `compile.rs`, `compile/mutual.rs`, and the anon `*_proj_addr` helpers to call them. - `verify_proj_addr_in_env` helper (#21). `ingress_anon_block` had the same "computed-address not in env" check repeated four times (DPrj/RPrj/IPrj/CPrj). DRY into one helper that produces a consistent error format. UX: - Anon display labels switched to `#<hex>` everywhere (#18). Rust was emitting `@<hex>` in progress/fail-out; Lean's `runCheckAnon` was emitting `#{i}` (result index). Standardize on `#<hex>` so the CLI failure summary is joinable with the fail-out file. This required exposing addresses per result slot to Lean — `rsCheckAnonFFI` now returns `Array (String × Option CheckError)` pairing each result with its content-address hex string. Rust builds the pairs via a new `build_anon_result_array` helper. - Stale "check-ixon" header in FailureLog (#16). One-line rename in the doc comment + `# ix check-ixon failures` writeln to `# ix check failures`. Tests: - `testPrimitivesParity` (PrimAddrs regen-parity test). Catches silent drift between hardcoded primitive addresses in `PrimAddrs::new()` and what `rsCompileEnvFFI` produces from the live Lean primitives. Plumbing: new `PrimAddrs::lean_parity_table()` returns `(lean_name, hex)` pairs; new `rs_prim_addrs_canonical` FFI exposes them to Lean; new test in `BuildPrimitives.lean` iterates `kernelPrimitives`, compares against the hardcoded table, and fails with a printable diff on mismatch (with instructions to regenerate). Skipped: `eagerReduce` — synthetic kernel marker whose PrimAddrs value (`0xff…3`) intentionally diverges from its compiled content hash (which collides with `id`). - 3 new corruption-detection tests in `serialize.rs` exercise the Section-2 verify check for `Env::get`, `Env::get_anon`, `Env::get_anon_mmap`. Verification: `cargo test --lib` 1025 passing (3 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe` 105487/105487 in ~21s, `--anon` 89010/89010 with peak RSS 1.4 GB, `.lake/build/bin/IxTests --ignored rust-kernel-build-primitives` shows both `build primitives dump` and `primitive address parity (PrimAddrs vs live compile)` pass. Out of scope (deferred to follow-ups): - #2 Format version bump (Env::FLAG) - #3 rs_kernel_check_consts_anon still uses Env::get on main - #8 Sealed marker trait for the lazy_anon transmute - #11/#12 AnonEnv audit (vestigial wrapper) - #13, #15, #17, #19, #20, #22-25 Various quality cleanups * Round-2 review fixes: mmap defense + small cleanup Six small items from the round-2 review of the anon-mode work: - N1: stat-at-open defense in `Env::get_anon_mmap` (`src/ix/ixon/serialize.rs`). Capture the file size before mmap; if the kernel's mapped length disagrees (file truncated between open and map) bail with a clear error instead of letting workers SIGBUS deep in `LazyConstant::get`. Truncate-in-place under a live mapping is still undefined per POSIX — documented as a caller contract — but the open-time check catches the common case. - New `get_anon_mmap_survives_file_unlink` test: load via mmap, unlink the path, then materialize both already-touched and not-yet-touched constants. Locks in the inode-retention invariant that the SIGBUS analysis depends on; a future refactor that switched to `mmap_anonymous` or copied bytes into a tmpfile would fail loudly here instead of letting workers SIGBUS in production. - #11: delete `AnonEnv::as_ixon_env_unchecked` (`src/ix/kernel/anon_env.rs`). Dead `pub(crate)` escape hatch marked `#[allow(dead_code)]` — confirmed zero callers and removed. - #13: `debug_assert!` on `OnceLock::set` for both result vectors (`src/ffi/kernel.rs`, meta + anon paths). The previous `let _ = results[idx].set(...)` silently dropped a re-set. If a future `build_*_work` dedup refactor breaks the one-write-per-slot invariant, debug builds now panic with the slot index instead of silently losing results. - #19: `allNames.contains` O(n²) preflight → `Std.HashSet` lookup (`Ix/Cli/CheckCmd.lean`). At mathlib scale (~700k env names × thousands of seed names) the previous linear scan-per-name spent measurable seconds on missing-name preflight alone. - N4: doc imprecision in `src/ix/ixon/lazy.rs` — the cache-policy preamble said the worker `KEnv` is "cleared between work items" but the actual cadence is `clear_every` items (`IX_KERNEL_CHECK_CLEAR_EVERY`, default 1 but tunable). Refined wording so the doc matches the code. Verification: `cargo test --lib` 1026 passing (1 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe --anon` 89010/89010 in 15.2s peak RSS 1.4 GB (regression-clean). Out of scope, separate follow-ups: - #2 Format version bump - #3 rs_kernel_check_consts_anon zkPCC FFI (Env::get → mmap) - #8 Sealed marker trait for the lazy_anon transmute - #10 child_arena closure side-effect (widen MField to tuple) - #12 AnonEnv duplicates bfs_refs / transitive_deps_excl - #14 anon worker bypasses M-generic display helpers - #17 Anon --fail-out docstring claims --consts-file compat - #20 Singleton-unwrap duplicated across compile paths - #22-24 Lean `partial def`, `.get!` in tests, unguarded as-u64 casts - N2/N3 Lean is_materialized() callers + compute_const_size_breakdown - Test gaps: rs_kernel_check_anon integration, concurrent mmap, verify_address-on-mmap-corruption, etc. * rustfmt * Tests: derive RawConst addrs from content (verify_address fix) The `Address::hash(bytes) == addr` defense added in f792102 rejects `RawConst { addr, const }` pairs where `addr` is uncorrelated with `serConstant const`. The Rust-side tests in `serialize.rs` were updated at the time (`store_canonical` helper), but three Lean-side sites still produced mismatched pairs and surface now as: × ∃: Env serialization Lean==Rust × ∃: serde RawEnv with data × ∃: serde RawEnv roundtrip deserialization failed: rs_de_env: Env::get: const at idx 0 bytes hash to 6110b739… but stored under b177ec1b… Three fixes, all derive `addr := Address.blake3 (serConstant c)`: - `Tests/Gen/Ixon.lean::genRawConst` — property-test generator was `RawConst.mk <$> genAddress <*> genConstant` (uncorrelated). Now generates the const first, then derives addr from `serConstant`. - `Tests/FFI/Lifecycle.lean::serdeTests` (`withData` unit case) — was using one `testAddr := Address.blake3 #[1,2,3]` for both the const and unrelated blob/comm slots. Split into `testAddr` (still used for the content-hash-free blob/comm) and `testConstAddr := Address.blake3 (serConstant testConst)`. Added a name entry for the new canonical addr. - `Tests/FFI/Lifecycle.lean::genSerdeRawEnv` — the "pool of addresses" pattern picked const addrs from a random pool that couldn't possibly match content hashes. Restructured: consts derive canonical addrs from content, each gets its own name-table entry appended to the pool-derived entries so the serde pipeline's "all addresses resolvable" invariant still holds. * Regenerate Aiur primitive addresses for singleton-unwrap Compiler's singleton non-inductive Muts unwrap (12630aa) changed the content hashes of 48 primitives (`Nat.rec`, `Nat.add`, `Nat.mul`, …). The Aiur kernel's `Ix/IxVM/Kernel/Primitive.lean` still held the pre-unwrap blake3 bytes, so primitive dispatch missed every affected constant and fell through to structural `Nat.rec` unfolding. Tests like `IxVMPrim.nat_mul_big` then drove millions of Nat.succ unfoldings inside the Aiur trace, OOMing the prover. Addresses regenerated from `PrimAddrs::lean_parity_table()` (which the branch had already updated in `src/ix/kernel/primitive.rs`). * Fix standalone Recr ingress: typ-based inductive lookup `find_matching_block_addr` heuristic in the standalone-Recr branch of `build_convert_inputs_walk` picks the inductive block by matching ctor count to rule count. When multiple in-scope inductives share the same ctor count, it returns the wrong block — the stored rules' `ctor_idx` then points to a sibling's ctor, while `populate_rules` (canonical) derives `ctor_idx` from the right inductive's `ctor_indices`. The mismatch surfaces as `compare_rules`'s `assert_eq!(s_ctor, c_ctor)` panic (e.g. `38 != 40` for `IxVMPrim.nat_land_lit`). Switch the standalone-Recr path to the same typ-based resolution `build_aux_recr_ctor_idxs` already uses for aux Recr blocks: peel `params + motives + minors + indices` foralls of `recr.typ`, take the major's head Ref, look it up in `refs` to get the inductive's address, then resolve `IPrj.block` for the Muts wrapper. Slice ctor positions for the specific member via `extract_member_ctor_idxs`. Also store the resolved Muts `block_addr` in `CKRecr` (was passing the heuristic's wrong address through to `convert_recursor`). --------- Co-authored-by: Arthur Paulino <arthurleonardo.ap@gmail.com> Co-authored-by: Samuel Burnham <45365069+samuelburnham@users.noreply.github.com>

plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed).

* Aiur Bytecode → Rust codegen for the IxVM kernel Adds a new Aiur pipeline stage that translates `Bytecode.Toplevel` into a Rust source module, one `fn aiur_fn_N` per Aiur function. The generated code mirrors `src/aiur/execute.rs`'s QueryRecord side effects exactly: same `function_queries.insert` timing, same cache-hit multiplicity bumps, same memory-queries insertion order, same `bytes{1,2}_queries` updates, same IO sequencing. Per-witness trace hashes match the interpreter byte-for-byte on every const tested (`Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). * `Ix/Aiur/Stages/Codegen.lean`: structured Rust IR (`RustExpr` / `RustStmt` / `RustItem` / `MatchArm` / labeled blocks) plus a formatter. Codegen is a structural walk over the Bytecode AST, no string templating. `EmitM = StateM EmitState` threads `(nextVal, nextLabel)` so per-ValIdx Rust locals and per-MC labels are fresh. * Aiur's `map: Vec<G>` is gone in the generated kernel: every ValIdx becomes a Rust local `__v_{i}: G`. Match-arm bodies snapshot `nextVal` on entry so per-arm allocations don't leak to siblings. `MatchContinue` / `Yield` use labeled-block + `break 'label [G; OUT_SIZE]` to bubble yielded values out and rebind them at outer scope. * Deep Aiur recursion uses `stacker::maybe_grow(64 KiB, 4 MiB, …)` at the top of every generated fn, so the native call stack grows on demand rather than pre-reserving a giant thread stack. Verified against shard 24 of the 64-way `init.ixes` partition, which SEGFAULTs without it. * `ix codegen`: new CLI command. Compiles the IxVM Aiur source, walks the bytecode, writes the generated Rust to a fixed path (`src/ix/aiur_ixvm.rs`). The output path is hard-coded; no override. * `src/ix/aiur_ixvm.rs`: the generated kernel, 743 `aiur_fn_*` + `execute_generated` dispatch. Auto-regenerated; do not edit. `#![cfg_attr(rustfmt, rustfmt::skip)]` + a broad `#![allow(unused_*, non_snake_case, clippy::all)]` since the layout is for the compiler, not humans. * `src/ix/aiur_ixvm_runner.rs`: `execute_ixvm(toplevel, fun_idx, args, io_buffer) -> Result<(QueryRecord, Vec<G>), ExecError>`. Same return shape as `Toplevel::execute`, but routes through `execute_generated`. * `src/aiur/synthesis.rs`: `AiurSystem::prove_ixvm(...)` — same shape as `prove`, but the execute step calls `execute_ixvm`. Verification-compatible (proofs from one path verify under the other). * `src/aiur/execute.rs`: helpers exposed for the codegen'd kernel (`bytes{1,2}_execute` → `pub(crate)`, `unconstrained_big_uint_div_mod_helper` extracted, `CodegenBytes{1,2}{Op,}` aliases re-exported, `QueryRecord::new` → `pub(crate)`, `ExecError::InvalidFunIdx` added). * `src/ffi/aiur/protocol.rs`: two new FFI exports — `rs_aiur_toplevel_execute_ixvm` and `rs_aiur_system_prove_ixvm`. Same wire format as the existing `_execute` / `prove` exports. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.executeIxVM` — mirror of `execute`. * `Ix/Aiur/Protocol.lean`: `AiurSystem.proveIxVM`. * `Ix/Cli/CheckCmd.lean`: `runCompiled` now dispatches through `executeIxVM`. The Rust bytecode interpreter is no longer reachable from `ix check` (the Lean-side `--interp` fallback stays for richer error diagnostics). * `Ix/Cli/ProveCmd.lean`: `proveOne` uses `proveIxVM`. * `Cargo.toml`: `stacker = "0.1"`. For witnesses where execute is the dominant work (single-const checks via `ix check 'X'`), the codegen'd kernel is ~1.6× faster than the bytecode interpreter end-to-end (measured on `Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). Tiny consts are within ~10% (stacker probe overhead amortises). Peak RSS is within a few MB of the interpreter — stacker grows the stack on demand, no giant upfront reservation. For shard-driven `ix check --shard K` runs the speedup is invisible: per-shard wall time is dominated (~92%) by `buildShardCheckEnvWitness` in Lean, NOT by the kernel execute (~8%). Cutting witness construction is the next lever. `Nat.add_comm` proof produced via `proveIxVM` verifies under the existing `AiurSystem::verify`. End-to-end: lake exe ix prove 'Nat.add_comm' → proof addr 0d0ab0f9… lake exe ix verify 0d0ab0f9… → ok * Move shard witness construction to Rust + parallelise Replaces `IxVM.ClaimHarness.buildShardCheckEnvWitness` (Lean side, ~92% of shard wall time on heavy partitions) with a Rust port that builds the `aiur::execute::IOBuffer` directly, without per-byte boxing into Lean `Aiur.G` values. The two hot phases run on rayon: * Closure walk: each owned addr's transitive `Constant.refs` + projection-block traversal runs on its own thread; results are deduped through a `dashmap::DashSet`. * Byte-to-G conversion: per-const `(key, data)` tuples are built in parallel chunks of 256; final IOBuffer assembly (channel arena append + key→`IOKeyInfo` map insert) runs serially because the arena index is monotonic. * `src/ix/aiur_ixvm_witness.rs` (new): `build_shard_check_env_witness` produces `(claim, claim_digest_input, io_buffer)` ready to feed to `execute_ixvm`. Mirrors the 6-channel layout documented in `Ix/IxVM/ClaimHarness.lean` (claim / asm tree / const bytes / Defn hint / blob discriminator / blob raw bytes). * `src/ffi/aiur/protocol.rs`: new FFI `rs_aiur_toplevel_shard_check_ixvm` bundles witness build + `execute_ixvm` into one cross-language trip. Returns the same `(output, ioBuffer, queryCounts)` shape as `rs_aiur_toplevel_execute_ixvm`, so the Lean shim stays drop-in compatible. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.shardCheckIxVM` wraps the FFI. * `Ix/Cli/CheckCmd.lean`: shard-mode `ix check` (`--ixe + --ixes`) now dispatches through `runShardOwnedNative` → `shardCheckIxVM`, bypassing the Lean witness builder. Single- shard and whole-partition paths share the fast route; the legacy `runShardCheckManifest` / `runShardCheckAll` callbacks stay live for `--interp` only. Shard 26 of the 64-way `init.ixes` partition, on the same host: | Variant | total | speedup | | --- | --: | --: | | Lean witness + bytecode-interp (baseline)| 1029 s | 1.0× | | Lean witness + codegen kernel | 1015 s | 1.01× | | Rust witness (serial) + codegen | 101 s | 9.95× | | Rust witness (parallel) + codegen | 80 s | 12.9× | The serial Rust witness collapses the 935 s of Lean per-byte boxing into ~23 s; rayon hides that 23 s under the codegen-kernel execute (~78 s) so witness build effectively becomes free at the shard level. FFT cost matches the bytecode interpreter exactly (107_006_963_281), so the QueryRecord layout is bit-identical across all four variants. * Closure walk uses single-source BFS with the global visited set shared via `DashSet::contains` checks before pushing to the per-thread stack — avoids redundant work without a per-owned union step. * Per-channel ordering: each chunk's partial `ChannelEntries` feeds the serial fold in iteration order, so channel arena contents match the Lean side's exactly. Test still relies on trace-hash parity from the earlier codegen commit, which exercised the same path through the bytecode interpreter and the codegen kernel. * Coverage check is currently skipped on the IxVM-native whole-partition path (`runShardManifestAllNative`); legacy `runShardCheckAll` still does it for `--interp`. Re-enable separately if needed. * Codegen: value-returning byte ops, dead-fold the unconstrained plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed). * Wire Rust witness fast path through every check/prove entry point Per-claim mode (a `Claim.check addr none`) builds a closure rooted at the target address. That closure can be the whole environment when the target is a heavily-shared root constant. The Lean witness builder paid per-byte boxing into `Aiur.G` for every byte in that closure — the same cost we already eliminated for shard mode. # Surface Five new FFIs, all bundling witness build + execute_ixvm (and STARK prove where applicable) into one cross-language trip so the `IOBuffer` never crosses the boundary mid-pipeline: * `rs_aiur_toplevel_check_addr_ixvm` — `Bytecode.Toplevel.checkAddrIxVM (toplevel) (funIdx) (ixePath) (addrBytes)`: per-claim check; builds witness for `Claim.check addr none` via `build_claim_check_witness`, runs `execute_ixvm`. Same return shape as `shardCheckIxVM`. * `rs_aiur_toplevel_check_env_bytes_ixvm` — same as above but takes the env as a serialized byte blob (`Ixon.serEnv`) instead of a `.ixe` path. Used by the compiled-Lean-env code path (`ix check NAME` without `--ixe`), where the env is built in Lean memory. * `rs_aiur_system_prove_addr_ixvm` — `AiurSystem.proveAddrIxVM (...)`: per-claim prove (witness + execute + STARK prove all in Rust). * `rs_aiur_system_prove_env_bytes_ixvm` — bytes-blob counterpart. * `rs_aiur_system_shard_prove_ixvm` — `AiurSystem.shardProveIxVM (...)`: per-shard prove end-to-end in Rust. `build_claim_check_witness` lives next to `build_shard_check_env_witness` in `src/ix/aiur_ixvm_witness.rs` and uses the same parallel closure walk + parallel byte→G conversion. The bytes-blob FFIs additionally harvest `anon_hints` from each `Def` named entry after `Env::get` — `Env::get` (full form, used by the bytes-blob path) doesn't populate `env.anon_hints` the way `Env::get_anon` does, but the kernel's `verify_claim` reads ch 3 (Defn reducibility hints) so the hints must end up in the env. This mirrors the harvest pass already inside `get_anon` at `src/ix/ixon/serialize.rs:1683`. # Wiring `Ix.Cli.CheckCmd.WitnessSource`: a three-arm discriminated union threaded through `forEachClaim` (the shared check/prove driver): * `.native ixePath addr` — `.ixe`-backed env, Rust mmap. * `.nativeBytes envBytes addr` — Lean-memory env serialized to bytes, Rust decodes via `Env::get`. * `.lean witness` — pre-built `ClaimWitness` (fallback). `runCompiled` and `proveOne` dispatch on the source. # Coverage | Command | Path | |---------------------------------------|-----------------------------------------------| | `ix check NAME --ixe` | **`checkAddrIxVM`** (new) | | `ix check NAME` (no `--ixe`) | **`checkEnvBytesIxVM`** (new) | | `ix check --claim hex --ixe` | `checkAddrIxVM` for `check addr none` | | `ix check --ixes --shard K` | `shardCheckIxVM` (existed) | | `ix check --ixes` | `shardCheckIxVM` (existed) | | `ix prove NAME --ixe` | **`proveAddrIxVM`** (new) | | `ix prove NAME` (no `--ixe`) | **`proveEnvBytesIxVM`** (new) | | `ix prove --ixes --shard K` | **`shardProveIxVM`** (new) | | `ix prove --ixes` | **`shardProveIxVM`** loop (new) | | `Benchmarks/Typecheck.lean` Phase 1 | `checkAddrIxVM` / `executeIxVM` | | `Benchmarks/Typecheck.lean` Phase 2 | `proveAddrIxVM` / `proveIxVM` | | `Benchmarks/IxVM.lean` | `proveIxVM` (was `prove`) | `--interp` route is preserved: it materialises any `WitnessSource` back into a `ClaimWitness` before driving the Aiur source interpreter. `--claim <hex>` over non-`check addr none` variants (`eval` / `reveal` / `contains` / `checkEnv-with-asm`) still uses the Lean witness builder. # Sanity * `ix check Nat.add_comm --ixe init.ixe` (warm): 3.1 s wall, FFT = 23_603_449. * `ix check Nat.add_comm` (compiled env via bytes blob, warm): 2.4 s wall, FFT = 23_603_449. # Known overheads (tracked for a follow-up redesign) * **Per-claim env re-parse on `--ixe + many names`**: each FFI call re-runs `Env::get_anon_mmap` on the same path. Mathlib-scale iteration pays the lazy-index build N times instead of once. * **`--interp + .nativeBytes` round-trip**: Lean serializes the compiled env then `materialise` deserializes it back, just to feed `runInterp`. * **`runShardProveNative` claim reconstruct**: redoes the closure walk + canonical `AssumptionTree` build Lean-side after `shardProveIxVM` already computed the same claim internally. A future `EnvHandle`-based API would close all three: build the env once into a Rust-owned handle, pass the handle to every per-claim/per-shard FFI, and have prove FFIs return the serialized claim bytes. * EnvHandle: Rust-owned env shared across per-claim / per-shard FFI calls Before: every per-claim and per-shard FFI re-parsed the Ixon env (`Env::get_anon_mmap` per call). On `--ixe + many names` / all-shards prove, this is the dominant overhead for iteration-heavy workflows — O(num_consts) lazy-index build × N targets. After: the env lives once per CLI invocation in a Rust-owned `EnvHandle`. Lean holds an opaque `Aiur.EnvHandle` reference and threads `@& EnvHandle` through every per-target FFI call. The env is parsed exactly once at handle construction; downstream calls share it. # Surface Five new FFIs collapse the previous six per-call variants (`checkAddrIxVM`, `checkEnvBytesIxVM`, `shardCheckIxVM`, `proveAddrIxVM`, `proveEnvBytesIxVM`, `shardProveIxVM` — all deleted): * `rs_aiur_env_handle_from_ixe(path) → LeanExternal<EnvHandle>`: mmap-load via `Env::get_anon_mmap`. Anon parser already harvests `anon_hints`; no post-pass. * `rs_aiur_env_handle_from_bytes(blob) → LeanExternal<EnvHandle>`: decode `Ixon.serEnv`-shape blob via `Env::get` + harvest `anon_hints` from each `Def` named entry. Used by the compiled-Lean-env path. * `rs_aiur_toplevel_check_addr_with_env(toplevel, fun_idx, handle, addr_bytes)`: per-claim check. Reuses handle's parsed env. * `rs_aiur_toplevel_shard_check_with_env(toplevel, fun_idx, handle, owned_blob)`: per-shard check. * `rs_aiur_system_prove_addr_with_env(system, fri, fun_idx, handle, addr_bytes) → (claim_bytes, proof, ioBuffer)`: per-claim prove. Rust serializes the reconstructed `Ix.Claim` via `ixon::Claim::put` so Lean can deserialize via `Ixon.runGet Ix.Claim.get` directly — no closure walk + canonical `AssumptionTree` recomputation Lean-side. * `rs_aiur_system_shard_prove_with_env(system, fri, fun_idx, handle, owned_blob) → (claim_bytes, proof, ioBuffer)`: per-shard prove. # Lean * `Aiur.EnvHandle` opaque type with `fromIxe` / `fromBytes`. * `Aiur.Bytecode.Toplevel.checkAddrWithEnv` / `shardCheckWithEnv`, `Aiur.AiurSystem.proveAddrWithEnv` / `shardProveWithEnv`. * `Ix.Cli.CheckCmd.Target` replaces `WitnessSource`: ```lean inductive Target where | addr (a : Address) -- Claim.check addr none | shard (owned : Array Address) -- Claim.checkEnv | leanW (w : ClaimWitness) -- --interp, --claim non-check ``` `runCompiled` / `proveOne` take `(envHandle?, target)`. The envHandle is `none` only for `.leanW` (`--interp` legacy path). * `runShardOwnedNative` and the manifest-driver helpers take the envHandle as a parameter. Single-shard mode builds it once; all-shards mode reuses the same handle across every shard's FFI call (eliminates per-shard re-mmap). * `runShardProveNative` deserializes the wire claim bytes via `Ixon.runGet Ix.Claim.get` instead of re-running `shardCheckEnvClaim`. # Benchmarks `Benchmarks/Typecheck.lean` builds the envHandle once before Phase 1, reuses it across Phase 1 (execute) + Phase 2 (prove). Both phases now go through `checkAddrWithEnv` / `proveAddrWithEnv` on the full-closure path. `--subject-only` still uses Lean `buildVerifyConst` + `executeIxVM` / `proveIxVM` (witnesses intentionally small there). # New crate `crates/ix/src/env_handle.rs` — the `EnvHandle` struct + constructors live in the existing `ix` crate next to the witness builder and codegen runner. # Sanity * Multi-target check (warm): `ix check --ixe init.ixe Nat.add_comm Nat.add` → 3.3 s wall, single envHandle shared across both targets. * Shard 26 check (warm): `ix check --ixe init.ixe --ixes init.ixes --shard 26` → ~78 s wall, FFT = 107_006_963_281 (parity preserved). # Coverage | Command | Path | |--------------------------------------|-----------------------------------| | `ix check NAME --ixe` | `checkAddrWithEnv` | | `ix check NAME` (no `--ixe`) | `checkAddrWithEnv` + fromBytes | | `ix check --claim hex --ixe` | `checkAddrWithEnv` for check-none | | `ix check --ixes --shard K` | `shardCheckWithEnv` | | `ix check --ixes` | `shardCheckWithEnv` × all shards | | `ix prove NAME --ixe` | `proveAddrWithEnv` | | `ix prove --ixes --shard K` | `shardProveWithEnv` | | `ix prove --ixes` | `shardProveWithEnv` × all shards | | `Benchmarks/Typecheck` Phase 1+2 | `checkAddrWithEnv` / `proveAddrWithEnv` | Only `--interp` and `--claim hex` over a non-`check addr none` persisted claim still build a Lean `ClaimWitness`; both go via the `.leanW` target arm. * ix codegen: add --check flag for CI; gate stale generated kernel `ix codegen --check` compares the emitted Rust source against the on-disk `crates/ix/src/aiur_ixvm.rs` and exits 0 if identical, 1 otherwise. No write side effect. ~2 s warm; fast enough to gate on every PR. Wired into the `lean-test` CI job so a forgotten regen on a kernel-touching PR fails CI instead of merging stale generated code that drifts from the Bytecode → Rust emitter. * ix check: --interp {source|bytecode} to bypass codegen kernel Two interpreter modes behind a single `--interp` flag: * `--interp source`: Aiur source interpreter (`Aiur.runFunction` over the source-level `Decls`). Richer per-step error diagnostics. * `--interp bytecode`: generic Aiur bytecode interpreter (`Bytecode.Toplevel.execute`, `rs_aiur_toplevel_execute` route). Skips the `ix codegen` + `cargo build --release` cycle needed after editing `Ix/IxVM/*.lean` — the bytecode is rebuilt Lean-side at exe load. Slower per-check than the codegen kernel; ideal for tight iteration on the IxVM source. * omit the flag entirely for the native codegen kernel (default). Invalid values (e.g. `--interp foo`) fail fast with a clear error. # Plumbing The `check_addr_with_env` and `shard_check_with_env` FFIs take a `use_bytecode: bool` and dispatch via a shared `dispatch_execute` helper. `--interp bytecode` sets it to `true`. The `.leanW` arm of `runCompiled` also picks between `Bytecode.Toplevel.execute` and `executeIxVM` based on the same flag. The `--interp source` path requires a Lean `ClaimWitness`. The existing driver had switched to `.addr`/`.shard` targets against the Rust-owned `EnvHandle`; source-interp had no way to materialise those. `forEachClaim` now takes a `forceLeanWitness : Bool` — when true, it builds a Lean witness for every target (via `mkWitness` / `loadIxonEnv` for the compiled-Lean-env path) and passes `.leanW` so `runInterp` can consume it directly. # Sanity * `ix check Std.Time.Week.Offset.ofMilliseconds` (warm): 9.3 s wall, FFT = 12_430_516_949 (codegen kernel). * `ix check --interp bytecode Std.Time.Week.Offset.ofMilliseconds` (warm): 14.6 s wall, FFT = 12_430_516_949 (bytecode interp). * `ix check --interp source Eq` (warm): 6.3 s wall, output `Eq: ()`. * `ix check --interp garbage Nat.add_comm`: exits 1 with clear error. * IxVM codegen: regen + rebase-fixups after KLevel port Aligns the ap/codegen-ixvm-native stack with main's KLevel pointer port (b84a500) and re-lands clippy-clean under the workspace lint set: - Regen crates/ix/src/aiur_ixvm.rs from Aiur source (KLevel = &KLevelNode changes the generated fn shapes; ~26k lines net churn). - Extend the generated file's #![allow(...)] header (via Ix/Aiur/Stages/Codegen.lean) with clippy::ptr_as_ptr, clippy::match_same_arms, and clippy::large_types_passed_by_value — three lints the codegen's straight-line style trips deterministically on every regen and that clippy::all doesn't cover. - rustfmt reflow in crates/aiur/src/execute.rs and a handful of ix / ffi files that CI now enforces via the ap/aiur-aux-recursor-parity toolchain. - Fix 4 real clippy warnings uncovered under the workspace lint set: * env_handle.rs: clone-on-Copy → deref (ReducibilityHints is Copy). * aiur_ixvm_witness.rs: collapse if-let-if, drop needless continue, slice::from_ref instead of &[x.clone()]. * aiur_ixvm_runner.rs: keep execute_ixvm's Vec<G> arg (required by AiurSystem::prove_ixvm's fn-pointer bound) + local #[allow( clippy::needless_pass_by_value)] with a comment explaining why. * ffi/aiur/protocol.rs: unqualify aiur::G/IOBuffer/QueryRecord after the workspace lint added unused_qualifications; is_multiple_of() over % 0. Verified: cargo clippy --workspace --all-targets --all-features --D warnings clean; lake test -- --ignored ixvm passes. * ix-check + ixvm tests: coverage gate, --interp source fix, codegen parity + rename ix crate to ixvm-codegen Four review fixes on the codegen-ixvm-native stack. - **Coverage gate on \`--ixes\` (all-shards, native path).** \`runShardManifestAllNative\` now calls \`shardsCover\` before running any shard, matching the pre-refactor \`runShardCheckAll\` behavior. Without this, a stale/truncated \`.ixes\` manifest makes \`ix check --ixes\` exit 0 while some env constants are never checked by any shard — the mundane trigger being: edit a definition, rebuild the \`.ixe\`, forget to re-run \`ix shard\`, and the constants that go unchecked are exactly the ones just edited. Single-shard \`--shard K\` path is unchanged (consistent with pre-refactor). \`shardsCover\` moved above \`runShardManifestAllNative\` to satisfy forward-decl ordering. - **\`ix check --ixe … --claim <hex> --interp source\` regression.** The \`claimHex\` arm in \`forEachClaim\` ignored \`forceLeanWitness\` and always mapped \`.check addr none\` claims to \`Target.addr\`. Under \`--interp source\`, \`runInterp\` then rejects \`.addr\` with "\`--interp requires a Lean witness; addr/shard targets unreachable here\`". Now mirrors the sibling name-based arms and routes through the Lean witness path when \`forceLeanWitness\` is on. Only bit the common claim shape. - **Codegen ↔ bytecode parity CI test.** New \`runParityCase\` + \`parityCases\` in \`Tests/Ix/IxVM.lean\` wire every constant already listed in \`kernelCheckEntries\` plus \`kernel_unit_tests\` through BOTH \`Toplevel.execute\` (bytecode) and \`Toplevel.executeIxVM\` (codegen'd Rust), and diff the returned \`(output, IOBuffer, QueryCounts)\` triples. Turns "generated kernel ≡ interpreter on QueryRecord" from reviewed-by-hand into checked-by-CI. 129 assertions run per suite invocation (43 constants × output + IOBuffer + QueryCount), all pass on this branch head. - **Rename \`crates/ix\` → \`crates/ixvm-codegen\`.** The old \`ix\` crate held only the codegen'd IxVM kernel + its runner + witness helpers. \`ix-kernel\` is the hand-written Rust ix typechecker; \`ix-common\` / \`ix-compile\` are Ix-level infrastructure. \`ixvm-codegen\` cleanly names what this crate holds. Path updates: workspace \`members\` and \`workspace.dependencies\`, ffi crate dep + \`use\` sites, Lean \`codegenOutPath\` in \`Ix/Cli/CodegenCmd.lean\`, doc reference in \`Ix/Aiur/Semantics/BytecodeFfi.lean\`. \`ix codegen\` now writes to \`crates/ixvm-codegen/src/aiur_ixvm.rs\`. Verified: \`cargo check --workspace\` + \`lake test -- --ignored ixvm\` both exit 0 post-rename; parity + pinned FFT tests all pass. * cargo-deny: allow ar_archive_writer's Apache-2.0 WITH LLVM-exception Post-rename, ixvm-codegen pulls stacker → psm → ar_archive_writer (a build-time transitive). ar_archive_writer is licensed under the LLVM-exception variant of Apache-2.0 (standard for LLVM tooling; no additional runtime obligation for downstream users), which isn't on the workspace allow list. Add a per-crate exception scoped to ar_archive_writer so we don't blanket-allow the license family across the tree.

arthurpaulino · 2025-02-04T13:52:08Z

No description provided.

Lays the groundwork for measuring the kernel performance plan (plans/okay-let-s-write-a-lucky-dolphin.md) per audit §10. Counters live in a new kernel::perf module; KEnv carries a PerfCounters field that is dumped from a Drop impl when IX_PERF_COUNTERS=1 is set. Unset is the production default — every increment short-circuits via a LazyLock<bool> so the cost is a single cached branch on the hot path. Wired into the cache get sites the audit identified: - whnf_cache hits/misses (whnf.rs around 201/211) - whnf_no_delta_cache hits/misses (whnf.rs around 437/446) - infer_cache and infer_only_cache hits/misses (infer.rs around 45/51) - def_eq_cache hits/misses (def_eq.rs around 137/149) - def_eq_failure set hits/inserts (def_eq.rs around 360) - per-constant peak/avg MAX_REC_FUEL consumption (recorded in TypeChecker::reset before the next constant resets it) Verified against lake test -- kernel-check-env --ignored: 192156/192156 constants pass in 251s (4% over the 242s baseline; overhead is the LazyLock branch + atomic load in the disabled path). P1: def_rank_id replaces def_weight_id for hint-priority comparison Audit Tier 1 #3 (kernel-perf-adversarial-audit-2026-04-26.md, §4.2): the prior u32 encoding mapped Abbrev to u32::MAX-1 and saturating-added Regular(h) to h+1, which collide at h ≥ u32::MAX-2. When that happens the delta-direction logic treats Abbrev and a maximally-heavy Regular as "same height" and unfolds both, instead of preferring Abbrev as Lean does (compare(d_t->get_hints(), d_s->get_hints()) at type_checker.cpp:910). Replace the u32 weight with a (class: u8, height: u32) tuple compared lexicographically: - Opaque / Theorem / unknown → (0, 0) - Regular(h) → (1, h) (height ordering preserved within class) - Abbrev → (2, 0) (strictly above every Regular) Update the two call sites (is_def_eq lazy-delta height comparison and lazy_delta_step). The map_or default for "missing head" is preserved as (u8::MAX, u32::MAX) — the branch is dead in practice (a_delta && b_delta imply both heads are present) but kept consistent with the prior u32::MAX sentinel. Two regression tests: - def_rank_abbrev_above_saturated_regular: Abbrev outranks Regular(u32::MAX) (the previous saturation collision). - def_rank_regular_orders_by_height: height monotonically orders Regular ranks within the class. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 256s (no regression vs the 242s pre-perf-counters baseline; the +14s is from the IX_PERF_COUNTERS=unset LazyLock branch added in the prior commit, not this change). P2: peel_proj_forall fast-paths syntactic Pi in projection inference Audit Tier 1 #2 (kernel-perf-adversarial-audit-2026-04-26.md, §7.2): infer_proj's two parameter-consuming loops (param peel and field peel) called self.whnf(&r)? unconditionally per iteration, on a body mutated by subst at the previous step. The whnf cache rarely hits between iterations and each call re-traverses the substituted body. Extract a peel_proj_forall(&r, err) helper that: - tries ExprData::All(..) syntactically first (no WHNF call), and - falls back to full self.whnf(e) only when the binder isn't already syntactic Pi. This mirrors Lean's inferProj at type_checker.cpp:582–610. Both projection-inference loops now call peel_proj_forall instead of unconditional whnf. Behaviorally equivalent — same WHNF semantics on miss, no semantic change otherwise. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 258s (parity with the post-pre-work, post-P1 baseline; no measurable regression and the cache-hit-rate counters will move on tactic-heavy workloads under IX_PERF_COUNTERS=1). P3a: WhnfFlags substrate (no behavior change) Lays the foundation for the Lean4Lean architectural alignment described in plans/okay-let-s-write-a-lucky-dolphin.md. Phase 3a is substrate-only — no call site is migrated to cheap mode yet, so behavior is unchanged. Adds: - WhnfFlags { cheap_rec, cheap_proj } with FULL and CHEAP consts and is_full(). CHEAP is currently equal to FULL until Phase 3c wires it. - whnf_core_with_flags (private): the existing whnf_core impl, now threading flags into recursive calls and try_iota_with_flags. - whnf_core / whnf_core_cheap (super): FULL/CHEAP wrappers. - whnf_no_delta_with_flags (private): the existing whnf_no_delta impl with the Prj branch gated on cheap_proj — falls back to full whnf on the projected value when not cheap. - whnf_no_delta (pub) / whnf_no_delta_cheap (super): wrappers. - try_iota_with_flags: gates major-premise WHNF and string-literal constructor reduction on cheap_rec. - try_proj_app_reduce_with_flags: gates projected-value WHNF on cheap_proj. Cache reads/writes (whnf_no_delta_cache, equiv-manager second-chance) are gated on flags.is_full(): cheap callers neither read nor write the cache, preserving the invariant that any cached entry is a fully-reduced normal form. Phase 3b will inline the projection branch into whnf_core to match Lean4Lean's two-layer architecture (refs/lean4lean/Lean4Lean/ TypeChecker.lean:266, 297). Phase 3c will flip CHEAP to enable cheap_proj and migrate specific def-eq sites. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 243s (matches the 242s baseline; substrate adds no measurable overhead when CHEAP == FULL). P3b: inline projection into whnf_core (Lean4Lean architectural alignment) Move the Prj branch from whnf_no_delta_with_flags into whnf_core_with_flags so our whnf_core matches Lean4Lean's whnfCore semantics exactly (refs/lean4lean/Lean4Lean/TypeChecker.lean:284-292, 337-341). Before this commit: whnf_core — beta + zeta + iota + cheap projection (recursive whnf_core on val, no delta) whnf_no_delta — whnf_core + FULL projection (full whnf on val) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta Lean4Lean's architecture has no whnf_no_delta layer. Their whnfCore includes projection, with the cheap_proj flag deciding whether the projected value uses whnfCore (cheap) or whnf (full). After this commit: whnf_core (with WhnfFlags) — beta + zeta + iota + projection (cheap_proj controls val reduction) whnf_no_delta — whnf_core(_, FULL) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta The bare-Prj branch in whnf_no_delta_with_flags is removed — whnf_core now handles it directly. The App-of-Prj branch stays in whnf_no_delta because whnf_core's loop returns once the outermost Prj is resolved; try_proj_app_reduce_with_flags gives one more attempt at the same cheap_proj policy when the outer expression is App(Prj, ...). Pure refactor, no semantic change with CHEAP == FULL. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 270s (within noise of the 243s pre-refactor baseline). Phase 3c will flip CHEAP to enable cheap_proj=true and migrate def-eq's lazy-delta sites surgically per Lean4Lean's pattern. P3c (postponed): document the HeaderParsedSnapshot regression P3a (substrate) and P3b (Lean4Lean architectural alignment) are committed. Phase 3c — flipping CHEAP to enable cheap_proj=true and migrating the def-eq lazy-delta sites — was attempted but reproduced 5 failures on chained projections in Lean.Language.Lean.HeaderParsedSnapshot.* even after P3b inlined the projection branch into whnf_core. The substrate is left in place; CHEAP stays equal to FULL until the regression's root cause is understood. Notes on the regression for the next investigator: - Failures: HeaderParsedSnapshot.{stx,result?,metaSnap,toSnapshot,ictx}, all with 'projection: type mismatch with declared struct'. - The struct `extends` a parent, so each projection is a chained Prj whose val is itself a Prj into the parent. - The error comes from infer.rs's infer_proj at the head-vs-struct_id address compare, after FULL whnf on val_ty. That whnf is FULL, but val_ty was inferred via paths that may have consulted a def-eq cache populated under cheap mode. Possible cache-poisoning suspect: def_eq_cache writes a `false` result for inputs whose lazy-delta loop bottomed out under cheap projections. The cache key uses raw a/b hashes, not cheap-reduced shapes, so a stored `false` is indistinguishable from a FULL `false` by future readers. - Lean4Lean does not have an analogous wide def_eq_cache; their failure cache is keyed only on same-spine pairs in lazyDeltaReductionStep. Future P3c iterations should either prove the cache poisoning theory incorrect or restrict def_eq_cache writes to FULL-derived results.

* Refactor Claim ADT + add merkle infrastructure for ZK aggregation Adds the format hooks needed before recursive verification lands in Aiur. Five-variant Claim ADT with explicit assumption commitments, a canonical Blake3 merkle module, a serializable AssumptionTree for recovering leaf sets, and a Contains claim for the discharge step. Claim ADT (5 variants): - Eval { input, output, assumptions: Option<Address> } - Check { const_addr, assumptions: Option<Address> } - CheckEnv { root, assumptions: Option<Address> } - Reveal { comm, info } -- unchanged, no assumptions - Contains { tree, const_addr } -- new, for inclusion proofs Tag4 reorganized to keep everything in single-byte tags: - 0xE for env, comm, AssumptionTree, and claims (slots 0-7) - 0xF for proofs (slots 0-4; 5-7 reserved) - Comm moved from variant 5 -> 1 Matches the "Variant (0-7)" constraint documented in docs/Ixon.md. Env serialization: - Every .ixe file now carries a canonical merkle root over its consts.keys() in the on-disk header (non-optional, 32 bytes; empty const sets use the zero-address sentinel). - Two envs with the same const set produce byte-identical roots regardless of insertion order. Verified on deserialize. New modules: - src/ix/ixon/merkle.rs + Ix/Merkle.lean: canonical sorted builder, free-form merkle_join composition, membership proofs, domain separation per RFC 6962. - src/ix/ixon/assumption_tree.rs + Ix/AssumptionTree.lean: serializable merkle tree with Leaf/Padding/Node variants. canonical() builds the same shape merkle_root_canonical hashes; join() is O(1) free-form composition. - src/ix/kernel/claim.rs: builders that compute transitive-dep assumptions from an env (build_check_claim, build_eval_claim, build_check_env_claim, env_merkle_root). Other: - Extracted shared BFS walker on Env::bfs_refs + Env::transitive_deps_excl; the inlined test-feature copy in lean_env.rs now calls into it. - Lean FFI export rs_env_merkle_root for cross-impl verification of the env root. - Proof bytes for all variants are uniform opaque ZK bytes; witness data (e.g., Contains merkle paths) is prover-side scratch consumed by the ZK circuit and not transmitted on the wire. - docs/Ixon.md Tag4 tables and env section updated; .ixe extension documented. Recursive verification (the ZK proof generation for Contains and the aggregation discharge transitions) is intentionally deferred to a follow-up. Tests: 993 Rust unit tests pass (was 953 pre-refactor), 813 Lean tests pass with no failures; cargo clippy clean. * Add lazy env deserialization and anon-mode kernel FFI Two related changes to reduce kernel memory + lay groundwork for metadata-isolated typechecking. **Lazy constant deserialization.** `Env::consts` now stores `LazyConstant` (`Arc<[u8]>` + `OnceLock<Arc<Constant>>`) instead of `Constant`. Constants are materialized on first access and the structured form is cached. Sparse access patterns (single-constant typecheck, transitive_deps walk, claim builders) only parse the closure they need; non-reachable constants stay as raw bytes. The `.ixe` section-2 layout gains a Tag0 length sidecar before each constant's Tag4 bytes: old: [addr:32] [Tag4 constant bytes] new: [addr:32] [Tag0 length] [Tag4 constant bytes] The length is section-level framing, not part of the constant's content hash. `Address::hash(raw_bytes) == addr` is preserved. `Constant::put`/`Constant::get` are unchanged. The Lean side (`Ix.Ixon.putEnv`/`getEnv`) reads/writes the new format. **Anon-mode kernel FFI.** New `rs_kernel_check_consts_anon(path, addrs, quiet)` exposes anonymous-mode typechecking by content address. The kernel runs as `KEnv<Anon>` / `TypeChecker<Anon>` with every `M::MField<T>` erased to `()`, so the typechecking logic structurally cannot read metadata. Useful for zkPCC verifiers that hold only addresses. Supporting pieces: - `KernelMode::HAS_META: bool` for future compile-time gating. - `AnonEnv<'a>` wrapper (`src/ix/kernel/anon_env.rs`) exposing only consts/blobs/transitive walks — no `named`/`names`/`comms`. - `Env::get_anon` reads header + blobs + consts, parse-and-drops metadata sections (3-5), returns an `Env` with empty `named`/ `names`/`comms`. Same merkle-root verification as `Env::get`. - `rs_de_env_anon` FFI + `Ix.Ixon.rsDeEnvAnon` Lean wrapper. - `Ix.KernelCheck.rsCheckConstsAnonFFI` Lean binding. Caveat: `ixon_ingress::<Anon>` still consults `Env::named` internally to enumerate work items. The resulting `KEnv<Anon>` is metadata-free so the typechecker is anon, but full ingress-level metadata isolation is a follow-up. Tests: 16 new (9 lazy + sparsity; 3 AnonEnv; 4 get_anon). All 1009 Rust unit tests pass; all 813 Lean tests pass; clippy clean. * Simplify ix check + add metadata-free anon mode - `ix check` becomes .ixe-only with a positional `<path>`. Drops the `--lean` compile-and-check flow and the `--env` flag; direct Lean → kernel checks remain available through `rsCheckConstsFFI` for tests. Removes the redundant `CheckIxonCmd.lean`. - New `--anon` flag runs a metadata-free kernel check: loads the .ixe via `Env::get_anon` (discards named/names/comms), enumerates work items from `env.consts` alone, and rebuilds member + ctor projection addresses deterministically via `Constant::commit`. Rejects `--consts`/`--ns`/`--consts-file` since it always checks everything. - Compiler: unwrap singleton non-inductive Muts blocks to standalone Defn/Recr. `Expr::Rec(0, univs)` already resolves correctly against a one-member block, so the wrapper was pure overhead; this keeps the env structurally uniform and matches `compile_single_def`. The anon ingress for standalones passes `mut_ctx_override = [self_id]` so self-recursive standalones still typecheck. - Kernel: anon parallel runner mirrors `run_checks_parallel_on_large _stacks` (32 workers, slow-detection, RSS tracking). Genericized `check_one_const<M>` / `check_consts_loop<M>` / `format_tc_error<M>` to share the runner. `KernelMode::meta_field_with` / `meta_field_try` gate metadata lookups in expression ingress at compile time. - Anon lazy ingress (`LazyAnonIngress` in tc.rs, helpers in ingress.rs) dedupes blocks via `kenv.blocks.contains_key(&KId<Anon>(B, ()))` before re-running `ingress_anon_block` (prevents N² re-ingestion when sibling projections fault separately within one worker check). - `Env::get_anon` now harvests `ReducibilityHints` from the otherwise-discarded Named section into a sidecar `Env::anon_hints: FxHashMap<Address, ReducibilityHints>`. Anon ingress threads these through a new `hints_override` parameter on `ingress_defn` so the lazy-delta tiebreak in `def_eq::def_rank_id` sees realistic heights instead of `Regular(0)`. Hints are performance advice, not correctness data — supplying them in anon mode preserves the metadata-free trust model. - Regenerate the canonical addresses in `PrimAddrs::new()`; singleton- unwrap changed the content hash of every self-recursive primitive (Nat.add, String, Nat.rec, …). End-to-end on compileinitstd.ixe (105,487 constants): meta: 105487/105487 in 20.6s, peak RSS 7.4GB anon: 89010/89010 in 17.8s, peak RSS 4.0GB * Clean up cargo clippy --all-targets - 11× `cloned_ref_to_slice_refs`: `&[x.clone()]` → `std::slice::from_ref(&x)` in `assumption_tree.rs` + `merkle.rs` test modules. Same allocator footprint, no clone, matches the lint's recommended idiom. - 2× `useless_vec`: `vec![0xE3, 0x00, 0x00]` → `[0xE3, 0x00, 0x00]` in serde-reject tests where the buffer is only borrowed. All 1011 unit tests pass. `cargo clippy --all-targets` is now clean. * ix check: print full content hashes + add --workers flag - The anon-mode progress / failure-log labels and the meta-mode hash-display fallback were truncating addresses to 16 hex chars (`@1f4b195aefa10e26`). Drop the `[..16]` slice so the full 64-char Blake3 hex is printed (`@1f4b195aefa10e2690d13c5b98b3d9124d3fbb5c…`), matching what tooling like `--fail-out` records and what the metadata-free workflow actually needs to identify a constant. - `--workers N` flag on `ix check` plumbs through the existing `IX_KERNEL_CHECK_WORKERS` env var that `resolve_kernel_check_workers` in `src/ffi/kernel.rs` reads. Useful for isolating per-worker memory cost (`--workers 1` lets you see the env's static footprint without per-worker overhead) and for capping concurrency in resource-tight contexts. * Mmap .ixe + cache-free LazyConstant for anon mode `lake exe ix check compilemathlib.ixe --anon` on a 3.2 GB env now peaks at ~11 GB RSS, down from ~40 GB; build_anon_work is 50× faster. Three intertwined changes: 1. Memory-map the .ixe (avoids ~3.2 GB heap copy of bytes) - Cargo: add memmap2 = "0.9". - `BytesSource` enum in `src/ix/ixon/lazy.rs` distinguishes heap-resident `Arc<[u8]>` from `(Arc<Mmap>, offset, len)` windows into a memory-mapped file. `LazyConstant::raw_bytes`, `verify_address`, `PartialEq` route through `BytesSource::as_slice` so every existing consumer is mmap-transparent. - `Env::get_anon_mmap(path)` in `src/ix/ixon/serialize.rs` opens the file, mmaps it, and stores Section-2 consts as mmap-backed `LazyConstant`s. Sections 3-4 (names + named) are still parsed transiently to harvest hints, then dropped before return. - `rs_kernel_check_anon` (`src/ffi/kernel.rs`) switches to `get_anon_mmap`; the old `std::fs::read` + `Env::get_anon` heap path is gone for the anon FFI. 2. Strip the persistent `LazyConstant` parsed-Constant cache - `LazyConstant.cache` was `Arc<OnceLock<Arc<Constant>>>` and accumulated forever — for mathlib that meant ~30 GB of parsed `Arc<Expr>` trees pinned in the env across the entire run. - New shape: `cache: Option<Arc<Constant>>`. Populated only by `from_constant` (compile-side, where we already own the parsed value). `from_bytes`/`from_mmap_slice` set `None`, and their `get()` parses fresh on every call without storing. - Re-parse cost is bounded by the existing per-worker dedup: `kenv.consts` already addresses each ingressed constant once per work item, and `clear_releasing_memory()` drops the kenv between items. The kenv is now the only persistent materialization layer. 3. `LazyConstant::peek_variant` for `build_anon_work` - One-byte read of the outer Tag4 head to identify the `ConstantInfo` variant — no body parse, no allocation. - `build_anon_work` now dispatches on `peek_variant()`; only `Muts` blocks trigger `lc.get()` (we need the member list for projection-address enumeration), and that `Arc<Constant>` drops at the end of the match arm. - Previously every constant was fully materialized at startup just to read its variant tag. With ~95% of the env being standalone/projection, the work-enumeration pass now skims the env at near-IO speed. Test updates: - `mmap_slice_roundtrips` (already added with the earlier mmap scaffolding) exercises the mmap window roundtrip. - New `from_bytes_does_not_cache` and `from_constant_clones_share_cache` document the new caching contract. - `peek_variant_*` tests cover every variant + empty-bytes and unknown-flag error paths. - `lazy_sparsity_only_materializes_closure` in `src/ix/ixon/env.rs` reframed to assert BFS-of-closure correctness instead of cache side-effects (`is_materialized()` no longer fires for lazy loads). End-to-end on compilemathlib.ixe (--anon, 32 workers): before: 640658/640658 in 266s, peak RSS ~40 GB after: 640658/640658 in 223s, peak RSS 11.3 GB * Address review findings: correctness + small refactors Code-review pass over the anon-mode / mmap / cache-strip work. This commit lands the highest-value subset — the correctness fixes and the small refactors that prevent future drift — and leaves the larger items (format-version bump, sealed marker trait, AnonEnv audit) for separate follow-ups. Correctness: - Verify per-constant address on load (#1). `LazyConstant::verify_address` existed but was never invoked from `Env::get`, `get_anon`, or `get_anon_mmap`. The env-level merkle root catches missing/extra entries but not byte-tampering of a constant whose key is intact; without this check, corruption surfaced much later inside `LazyConstant::get` with a misleading parse error. Inline the `Address::hash(bytes) == addr` check in each loader's Section-2 loop. Added 3 corruption-detection tests (`env_const_bytes_tampering_*`). This required updating a handful of existing tests that stored constants under fake `Address::hash(b"a")` keys instead of their true content hashes — round-tripping such envs now correctly rejects. Added a `store_canonical(env, c) -> Address` test helper for the canonical pattern, and `*_discriminator(refs, n)` so tests can produce content-distinct constants when the same ref-set would otherwise collide. - Hard-error in `ixon_env_to_decoded` on parse failure (#6). `src/ffi/ixon/env.rs` used `filter_map` to silently drop any const whose bytes failed `LazyConstant::get` — the Lean caller had no signal of the lost entries. Switch to a Result-collecting loop and propagate the first parse error; update both FFI call sites (`rs_de_env`, `rs_de_env_anon`). - Doc fixes (#4, #5). `docs/Ixon.md`'s AssumptionTree section described only `Leaf=0x00` and `Node=0x01`, missing `Padding=0x01` (and the docs' `Node` tag collided with the real `Padding` tag — the real `Node` is `0x02`). Also: the env-section table cell said "opt-tagged merkle root" but the implementation writes a bare 32-byte address. Refactors that prevent future drift: - Canonical projection-address helpers (#9). Four production sites reconstructed `Constant::new(IxonCI::{D,I,R,C}Prj{...}).commit().0` by hand; the anon pipeline silently breaks if any one drifts. Extract `defn_proj_address` / `indc_proj_address` / `recr_proj_address` / `ctor_proj_address` (plus `_constant` variants) in `src/ix/ixon/constant.rs`. Update `compile.rs`, `compile/mutual.rs`, and the anon `*_proj_addr` helpers to call them. - `verify_proj_addr_in_env` helper (#21). `ingress_anon_block` had the same "computed-address not in env" check repeated four times (DPrj/RPrj/IPrj/CPrj). DRY into one helper that produces a consistent error format. UX: - Anon display labels switched to `#<hex>` everywhere (#18). Rust was emitting `@<hex>` in progress/fail-out; Lean's `runCheckAnon` was emitting `#{i}` (result index). Standardize on `#<hex>` so the CLI failure summary is joinable with the fail-out file. This required exposing addresses per result slot to Lean — `rsCheckAnonFFI` now returns `Array (String × Option CheckError)` pairing each result with its content-address hex string. Rust builds the pairs via a new `build_anon_result_array` helper. - Stale "check-ixon" header in FailureLog (#16). One-line rename in the doc comment + `# ix check-ixon failures` writeln to `# ix check failures`. Tests: - `testPrimitivesParity` (PrimAddrs regen-parity test). Catches silent drift between hardcoded primitive addresses in `PrimAddrs::new()` and what `rsCompileEnvFFI` produces from the live Lean primitives. Plumbing: new `PrimAddrs::lean_parity_table()` returns `(lean_name, hex)` pairs; new `rs_prim_addrs_canonical` FFI exposes them to Lean; new test in `BuildPrimitives.lean` iterates `kernelPrimitives`, compares against the hardcoded table, and fails with a printable diff on mismatch (with instructions to regenerate). Skipped: `eagerReduce` — synthetic kernel marker whose PrimAddrs value (`0xff…3`) intentionally diverges from its compiled content hash (which collides with `id`). - 3 new corruption-detection tests in `serialize.rs` exercise the Section-2 verify check for `Env::get`, `Env::get_anon`, `Env::get_anon_mmap`. Verification: `cargo test --lib` 1025 passing (3 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe` 105487/105487 in ~21s, `--anon` 89010/89010 with peak RSS 1.4 GB, `.lake/build/bin/IxTests --ignored rust-kernel-build-primitives` shows both `build primitives dump` and `primitive address parity (PrimAddrs vs live compile)` pass. Out of scope (deferred to follow-ups): - #2 Format version bump (Env::FLAG) - #3 rs_kernel_check_consts_anon still uses Env::get on main - #8 Sealed marker trait for the lazy_anon transmute - #11/#12 AnonEnv audit (vestigial wrapper) - #13, #15, #17, #19, #20, #22-25 Various quality cleanups * Round-2 review fixes: mmap defense + small cleanup Six small items from the round-2 review of the anon-mode work: - N1: stat-at-open defense in `Env::get_anon_mmap` (`src/ix/ixon/serialize.rs`). Capture the file size before mmap; if the kernel's mapped length disagrees (file truncated between open and map) bail with a clear error instead of letting workers SIGBUS deep in `LazyConstant::get`. Truncate-in-place under a live mapping is still undefined per POSIX — documented as a caller contract — but the open-time check catches the common case. - New `get_anon_mmap_survives_file_unlink` test: load via mmap, unlink the path, then materialize both already-touched and not-yet-touched constants. Locks in the inode-retention invariant that the SIGBUS analysis depends on; a future refactor that switched to `mmap_anonymous` or copied bytes into a tmpfile would fail loudly here instead of letting workers SIGBUS in production. - #11: delete `AnonEnv::as_ixon_env_unchecked` (`src/ix/kernel/anon_env.rs`). Dead `pub(crate)` escape hatch marked `#[allow(dead_code)]` — confirmed zero callers and removed. - #13: `debug_assert!` on `OnceLock::set` for both result vectors (`src/ffi/kernel.rs`, meta + anon paths). The previous `let _ = results[idx].set(...)` silently dropped a re-set. If a future `build_*_work` dedup refactor breaks the one-write-per-slot invariant, debug builds now panic with the slot index instead of silently losing results. - #19: `allNames.contains` O(n²) preflight → `Std.HashSet` lookup (`Ix/Cli/CheckCmd.lean`). At mathlib scale (~700k env names × thousands of seed names) the previous linear scan-per-name spent measurable seconds on missing-name preflight alone. - N4: doc imprecision in `src/ix/ixon/lazy.rs` — the cache-policy preamble said the worker `KEnv` is "cleared between work items" but the actual cadence is `clear_every` items (`IX_KERNEL_CHECK_CLEAR_EVERY`, default 1 but tunable). Refined wording so the doc matches the code. Verification: `cargo test --lib` 1026 passing (1 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe --anon` 89010/89010 in 15.2s peak RSS 1.4 GB (regression-clean). Out of scope, separate follow-ups: - #2 Format version bump - #3 rs_kernel_check_consts_anon zkPCC FFI (Env::get → mmap) - #8 Sealed marker trait for the lazy_anon transmute - #10 child_arena closure side-effect (widen MField to tuple) - #12 AnonEnv duplicates bfs_refs / transitive_deps_excl - #14 anon worker bypasses M-generic display helpers - #17 Anon --fail-out docstring claims --consts-file compat - #20 Singleton-unwrap duplicated across compile paths - #22-24 Lean `partial def`, `.get!` in tests, unguarded as-u64 casts - N2/N3 Lean is_materialized() callers + compute_const_size_breakdown - Test gaps: rs_kernel_check_anon integration, concurrent mmap, verify_address-on-mmap-corruption, etc. * rustfmt * Tests: derive RawConst addrs from content (verify_address fix) The `Address::hash(bytes) == addr` defense added in f792102 rejects `RawConst { addr, const }` pairs where `addr` is uncorrelated with `serConstant const`. The Rust-side tests in `serialize.rs` were updated at the time (`store_canonical` helper), but three Lean-side sites still produced mismatched pairs and surface now as: × ∃: Env serialization Lean==Rust × ∃: serde RawEnv with data × ∃: serde RawEnv roundtrip deserialization failed: rs_de_env: Env::get: const at idx 0 bytes hash to 6110b739… but stored under b177ec1b… Three fixes, all derive `addr := Address.blake3 (serConstant c)`: - `Tests/Gen/Ixon.lean::genRawConst` — property-test generator was `RawConst.mk <$> genAddress <*> genConstant` (uncorrelated). Now generates the const first, then derives addr from `serConstant`. - `Tests/FFI/Lifecycle.lean::serdeTests` (`withData` unit case) — was using one `testAddr := Address.blake3 #[1,2,3]` for both the const and unrelated blob/comm slots. Split into `testAddr` (still used for the content-hash-free blob/comm) and `testConstAddr := Address.blake3 (serConstant testConst)`. Added a name entry for the new canonical addr. - `Tests/FFI/Lifecycle.lean::genSerdeRawEnv` — the "pool of addresses" pattern picked const addrs from a random pool that couldn't possibly match content hashes. Restructured: consts derive canonical addrs from content, each gets its own name-table entry appended to the pool-derived entries so the serde pipeline's "all addresses resolvable" invariant still holds. * Regenerate Aiur primitive addresses for singleton-unwrap Compiler's singleton non-inductive Muts unwrap (12630aa) changed the content hashes of 48 primitives (`Nat.rec`, `Nat.add`, `Nat.mul`, …). The Aiur kernel's `Ix/IxVM/Kernel/Primitive.lean` still held the pre-unwrap blake3 bytes, so primitive dispatch missed every affected constant and fell through to structural `Nat.rec` unfolding. Tests like `IxVMPrim.nat_mul_big` then drove millions of Nat.succ unfoldings inside the Aiur trace, OOMing the prover. Addresses regenerated from `PrimAddrs::lean_parity_table()` (which the branch had already updated in `src/ix/kernel/primitive.rs`). * Fix standalone Recr ingress: typ-based inductive lookup `find_matching_block_addr` heuristic in the standalone-Recr branch of `build_convert_inputs_walk` picks the inductive block by matching ctor count to rule count. When multiple in-scope inductives share the same ctor count, it returns the wrong block — the stored rules' `ctor_idx` then points to a sibling's ctor, while `populate_rules` (canonical) derives `ctor_idx` from the right inductive's `ctor_indices`. The mismatch surfaces as `compare_rules`'s `assert_eq!(s_ctor, c_ctor)` panic (e.g. `38 != 40` for `IxVMPrim.nat_land_lit`). Switch the standalone-Recr path to the same typ-based resolution `build_aux_recr_ctor_idxs` already uses for aux Recr blocks: peel `params + motives + minors + indices` foralls of `recr.typ`, take the major's head Ref, look it up in `refs` to get the inductive's address, then resolve `IPrj.block` for the Muts wrapper. Slice ctor positions for the specific member via `extract_member_ctor_idxs`. Also store the resolved Muts `block_addr` in `CKRecr` (was passing the heuristic's wrong address through to `convert_recursor`). --------- Co-authored-by: Arthur Paulino <arthurleonardo.ap@gmail.com> Co-authored-by: Samuel Burnham <45365069+samuelburnham@users.noreply.github.com>

plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed).

* Aiur Bytecode → Rust codegen for the IxVM kernel Adds a new Aiur pipeline stage that translates `Bytecode.Toplevel` into a Rust source module, one `fn aiur_fn_N` per Aiur function. The generated code mirrors `src/aiur/execute.rs`'s QueryRecord side effects exactly: same `function_queries.insert` timing, same cache-hit multiplicity bumps, same memory-queries insertion order, same `bytes{1,2}_queries` updates, same IO sequencing. Per-witness trace hashes match the interpreter byte-for-byte on every const tested (`Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). * `Ix/Aiur/Stages/Codegen.lean`: structured Rust IR (`RustExpr` / `RustStmt` / `RustItem` / `MatchArm` / labeled blocks) plus a formatter. Codegen is a structural walk over the Bytecode AST, no string templating. `EmitM = StateM EmitState` threads `(nextVal, nextLabel)` so per-ValIdx Rust locals and per-MC labels are fresh. * Aiur's `map: Vec<G>` is gone in the generated kernel: every ValIdx becomes a Rust local `__v_{i}: G`. Match-arm bodies snapshot `nextVal` on entry so per-arm allocations don't leak to siblings. `MatchContinue` / `Yield` use labeled-block + `break 'label [G; OUT_SIZE]` to bubble yielded values out and rebind them at outer scope. * Deep Aiur recursion uses `stacker::maybe_grow(64 KiB, 4 MiB, …)` at the top of every generated fn, so the native call stack grows on demand rather than pre-reserving a giant thread stack. Verified against shard 24 of the 64-way `init.ixes` partition, which SEGFAULTs without it. * `ix codegen`: new CLI command. Compiles the IxVM Aiur source, walks the bytecode, writes the generated Rust to a fixed path (`src/ix/aiur_ixvm.rs`). The output path is hard-coded; no override. * `src/ix/aiur_ixvm.rs`: the generated kernel, 743 `aiur_fn_*` + `execute_generated` dispatch. Auto-regenerated; do not edit. `#![cfg_attr(rustfmt, rustfmt::skip)]` + a broad `#![allow(unused_*, non_snake_case, clippy::all)]` since the layout is for the compiler, not humans. * `src/ix/aiur_ixvm_runner.rs`: `execute_ixvm(toplevel, fun_idx, args, io_buffer) -> Result<(QueryRecord, Vec<G>), ExecError>`. Same return shape as `Toplevel::execute`, but routes through `execute_generated`. * `src/aiur/synthesis.rs`: `AiurSystem::prove_ixvm(...)` — same shape as `prove`, but the execute step calls `execute_ixvm`. Verification-compatible (proofs from one path verify under the other). * `src/aiur/execute.rs`: helpers exposed for the codegen'd kernel (`bytes{1,2}_execute` → `pub(crate)`, `unconstrained_big_uint_div_mod_helper` extracted, `CodegenBytes{1,2}{Op,}` aliases re-exported, `QueryRecord::new` → `pub(crate)`, `ExecError::InvalidFunIdx` added). * `src/ffi/aiur/protocol.rs`: two new FFI exports — `rs_aiur_toplevel_execute_ixvm` and `rs_aiur_system_prove_ixvm`. Same wire format as the existing `_execute` / `prove` exports. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.executeIxVM` — mirror of `execute`. * `Ix/Aiur/Protocol.lean`: `AiurSystem.proveIxVM`. * `Ix/Cli/CheckCmd.lean`: `runCompiled` now dispatches through `executeIxVM`. The Rust bytecode interpreter is no longer reachable from `ix check` (the Lean-side `--interp` fallback stays for richer error diagnostics). * `Ix/Cli/ProveCmd.lean`: `proveOne` uses `proveIxVM`. * `Cargo.toml`: `stacker = "0.1"`. For witnesses where execute is the dominant work (single-const checks via `ix check 'X'`), the codegen'd kernel is ~1.6× faster than the bytecode interpreter end-to-end (measured on `Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). Tiny consts are within ~10% (stacker probe overhead amortises). Peak RSS is within a few MB of the interpreter — stacker grows the stack on demand, no giant upfront reservation. For shard-driven `ix check --shard K` runs the speedup is invisible: per-shard wall time is dominated (~92%) by `buildShardCheckEnvWitness` in Lean, NOT by the kernel execute (~8%). Cutting witness construction is the next lever. `Nat.add_comm` proof produced via `proveIxVM` verifies under the existing `AiurSystem::verify`. End-to-end: lake exe ix prove 'Nat.add_comm' → proof addr 0d0ab0f9… lake exe ix verify 0d0ab0f9… → ok * Move shard witness construction to Rust + parallelise Replaces `IxVM.ClaimHarness.buildShardCheckEnvWitness` (Lean side, ~92% of shard wall time on heavy partitions) with a Rust port that builds the `aiur::execute::IOBuffer` directly, without per-byte boxing into Lean `Aiur.G` values. The two hot phases run on rayon: * Closure walk: each owned addr's transitive `Constant.refs` + projection-block traversal runs on its own thread; results are deduped through a `dashmap::DashSet`. * Byte-to-G conversion: per-const `(key, data)` tuples are built in parallel chunks of 256; final IOBuffer assembly (channel arena append + key→`IOKeyInfo` map insert) runs serially because the arena index is monotonic. * `src/ix/aiur_ixvm_witness.rs` (new): `build_shard_check_env_witness` produces `(claim, claim_digest_input, io_buffer)` ready to feed to `execute_ixvm`. Mirrors the 6-channel layout documented in `Ix/IxVM/ClaimHarness.lean` (claim / asm tree / const bytes / Defn hint / blob discriminator / blob raw bytes). * `src/ffi/aiur/protocol.rs`: new FFI `rs_aiur_toplevel_shard_check_ixvm` bundles witness build + `execute_ixvm` into one cross-language trip. Returns the same `(output, ioBuffer, queryCounts)` shape as `rs_aiur_toplevel_execute_ixvm`, so the Lean shim stays drop-in compatible. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.shardCheckIxVM` wraps the FFI. * `Ix/Cli/CheckCmd.lean`: shard-mode `ix check` (`--ixe + --ixes`) now dispatches through `runShardOwnedNative` → `shardCheckIxVM`, bypassing the Lean witness builder. Single- shard and whole-partition paths share the fast route; the legacy `runShardCheckManifest` / `runShardCheckAll` callbacks stay live for `--interp` only. Shard 26 of the 64-way `init.ixes` partition, on the same host: | Variant | total | speedup | | --- | --: | --: | | Lean witness + bytecode-interp (baseline)| 1029 s | 1.0× | | Lean witness + codegen kernel | 1015 s | 1.01× | | Rust witness (serial) + codegen | 101 s | 9.95× | | Rust witness (parallel) + codegen | 80 s | 12.9× | The serial Rust witness collapses the 935 s of Lean per-byte boxing into ~23 s; rayon hides that 23 s under the codegen-kernel execute (~78 s) so witness build effectively becomes free at the shard level. FFT cost matches the bytecode interpreter exactly (107_006_963_281), so the QueryRecord layout is bit-identical across all four variants. * Closure walk uses single-source BFS with the global visited set shared via `DashSet::contains` checks before pushing to the per-thread stack — avoids redundant work without a per-owned union step. * Per-channel ordering: each chunk's partial `ChannelEntries` feeds the serial fold in iteration order, so channel arena contents match the Lean side's exactly. Test still relies on trace-hash parity from the earlier codegen commit, which exercised the same path through the bytecode interpreter and the codegen kernel. * Coverage check is currently skipped on the IxVM-native whole-partition path (`runShardManifestAllNative`); legacy `runShardCheckAll` still does it for `--interp`. Re-enable separately if needed. * Codegen: value-returning byte ops, dead-fold the unconstrained plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed). * Wire Rust witness fast path through every check/prove entry point Per-claim mode (a `Claim.check addr none`) builds a closure rooted at the target address. That closure can be the whole environment when the target is a heavily-shared root constant. The Lean witness builder paid per-byte boxing into `Aiur.G` for every byte in that closure — the same cost we already eliminated for shard mode. # Surface Five new FFIs, all bundling witness build + execute_ixvm (and STARK prove where applicable) into one cross-language trip so the `IOBuffer` never crosses the boundary mid-pipeline: * `rs_aiur_toplevel_check_addr_ixvm` — `Bytecode.Toplevel.checkAddrIxVM (toplevel) (funIdx) (ixePath) (addrBytes)`: per-claim check; builds witness for `Claim.check addr none` via `build_claim_check_witness`, runs `execute_ixvm`. Same return shape as `shardCheckIxVM`. * `rs_aiur_toplevel_check_env_bytes_ixvm` — same as above but takes the env as a serialized byte blob (`Ixon.serEnv`) instead of a `.ixe` path. Used by the compiled-Lean-env code path (`ix check NAME` without `--ixe`), where the env is built in Lean memory. * `rs_aiur_system_prove_addr_ixvm` — `AiurSystem.proveAddrIxVM (...)`: per-claim prove (witness + execute + STARK prove all in Rust). * `rs_aiur_system_prove_env_bytes_ixvm` — bytes-blob counterpart. * `rs_aiur_system_shard_prove_ixvm` — `AiurSystem.shardProveIxVM (...)`: per-shard prove end-to-end in Rust. `build_claim_check_witness` lives next to `build_shard_check_env_witness` in `src/ix/aiur_ixvm_witness.rs` and uses the same parallel closure walk + parallel byte→G conversion. The bytes-blob FFIs additionally harvest `anon_hints` from each `Def` named entry after `Env::get` — `Env::get` (full form, used by the bytes-blob path) doesn't populate `env.anon_hints` the way `Env::get_anon` does, but the kernel's `verify_claim` reads ch 3 (Defn reducibility hints) so the hints must end up in the env. This mirrors the harvest pass already inside `get_anon` at `src/ix/ixon/serialize.rs:1683`. # Wiring `Ix.Cli.CheckCmd.WitnessSource`: a three-arm discriminated union threaded through `forEachClaim` (the shared check/prove driver): * `.native ixePath addr` — `.ixe`-backed env, Rust mmap. * `.nativeBytes envBytes addr` — Lean-memory env serialized to bytes, Rust decodes via `Env::get`. * `.lean witness` — pre-built `ClaimWitness` (fallback). `runCompiled` and `proveOne` dispatch on the source. # Coverage | Command | Path | |---------------------------------------|-----------------------------------------------| | `ix check NAME --ixe` | **`checkAddrIxVM`** (new) | | `ix check NAME` (no `--ixe`) | **`checkEnvBytesIxVM`** (new) | | `ix check --claim hex --ixe` | `checkAddrIxVM` for `check addr none` | | `ix check --ixes --shard K` | `shardCheckIxVM` (existed) | | `ix check --ixes` | `shardCheckIxVM` (existed) | | `ix prove NAME --ixe` | **`proveAddrIxVM`** (new) | | `ix prove NAME` (no `--ixe`) | **`proveEnvBytesIxVM`** (new) | | `ix prove --ixes --shard K` | **`shardProveIxVM`** (new) | | `ix prove --ixes` | **`shardProveIxVM`** loop (new) | | `Benchmarks/Typecheck.lean` Phase 1 | `checkAddrIxVM` / `executeIxVM` | | `Benchmarks/Typecheck.lean` Phase 2 | `proveAddrIxVM` / `proveIxVM` | | `Benchmarks/IxVM.lean` | `proveIxVM` (was `prove`) | `--interp` route is preserved: it materialises any `WitnessSource` back into a `ClaimWitness` before driving the Aiur source interpreter. `--claim <hex>` over non-`check addr none` variants (`eval` / `reveal` / `contains` / `checkEnv-with-asm`) still uses the Lean witness builder. # Sanity * `ix check Nat.add_comm --ixe init.ixe` (warm): 3.1 s wall, FFT = 23_603_449. * `ix check Nat.add_comm` (compiled env via bytes blob, warm): 2.4 s wall, FFT = 23_603_449. # Known overheads (tracked for a follow-up redesign) * **Per-claim env re-parse on `--ixe + many names`**: each FFI call re-runs `Env::get_anon_mmap` on the same path. Mathlib-scale iteration pays the lazy-index build N times instead of once. * **`--interp + .nativeBytes` round-trip**: Lean serializes the compiled env then `materialise` deserializes it back, just to feed `runInterp`. * **`runShardProveNative` claim reconstruct**: redoes the closure walk + canonical `AssumptionTree` build Lean-side after `shardProveIxVM` already computed the same claim internally. A future `EnvHandle`-based API would close all three: build the env once into a Rust-owned handle, pass the handle to every per-claim/per-shard FFI, and have prove FFIs return the serialized claim bytes. * EnvHandle: Rust-owned env shared across per-claim / per-shard FFI calls Before: every per-claim and per-shard FFI re-parsed the Ixon env (`Env::get_anon_mmap` per call). On `--ixe + many names` / all-shards prove, this is the dominant overhead for iteration-heavy workflows — O(num_consts) lazy-index build × N targets. After: the env lives once per CLI invocation in a Rust-owned `EnvHandle`. Lean holds an opaque `Aiur.EnvHandle` reference and threads `@& EnvHandle` through every per-target FFI call. The env is parsed exactly once at handle construction; downstream calls share it. # Surface Five new FFIs collapse the previous six per-call variants (`checkAddrIxVM`, `checkEnvBytesIxVM`, `shardCheckIxVM`, `proveAddrIxVM`, `proveEnvBytesIxVM`, `shardProveIxVM` — all deleted): * `rs_aiur_env_handle_from_ixe(path) → LeanExternal<EnvHandle>`: mmap-load via `Env::get_anon_mmap`. Anon parser already harvests `anon_hints`; no post-pass. * `rs_aiur_env_handle_from_bytes(blob) → LeanExternal<EnvHandle>`: decode `Ixon.serEnv`-shape blob via `Env::get` + harvest `anon_hints` from each `Def` named entry. Used by the compiled-Lean-env path. * `rs_aiur_toplevel_check_addr_with_env(toplevel, fun_idx, handle, addr_bytes)`: per-claim check. Reuses handle's parsed env. * `rs_aiur_toplevel_shard_check_with_env(toplevel, fun_idx, handle, owned_blob)`: per-shard check. * `rs_aiur_system_prove_addr_with_env(system, fri, fun_idx, handle, addr_bytes) → (claim_bytes, proof, ioBuffer)`: per-claim prove. Rust serializes the reconstructed `Ix.Claim` via `ixon::Claim::put` so Lean can deserialize via `Ixon.runGet Ix.Claim.get` directly — no closure walk + canonical `AssumptionTree` recomputation Lean-side. * `rs_aiur_system_shard_prove_with_env(system, fri, fun_idx, handle, owned_blob) → (claim_bytes, proof, ioBuffer)`: per-shard prove. # Lean * `Aiur.EnvHandle` opaque type with `fromIxe` / `fromBytes`. * `Aiur.Bytecode.Toplevel.checkAddrWithEnv` / `shardCheckWithEnv`, `Aiur.AiurSystem.proveAddrWithEnv` / `shardProveWithEnv`. * `Ix.Cli.CheckCmd.Target` replaces `WitnessSource`: ```lean inductive Target where | addr (a : Address) -- Claim.check addr none | shard (owned : Array Address) -- Claim.checkEnv | leanW (w : ClaimWitness) -- --interp, --claim non-check ``` `runCompiled` / `proveOne` take `(envHandle?, target)`. The envHandle is `none` only for `.leanW` (`--interp` legacy path). * `runShardOwnedNative` and the manifest-driver helpers take the envHandle as a parameter. Single-shard mode builds it once; all-shards mode reuses the same handle across every shard's FFI call (eliminates per-shard re-mmap). * `runShardProveNative` deserializes the wire claim bytes via `Ixon.runGet Ix.Claim.get` instead of re-running `shardCheckEnvClaim`. # Benchmarks `Benchmarks/Typecheck.lean` builds the envHandle once before Phase 1, reuses it across Phase 1 (execute) + Phase 2 (prove). Both phases now go through `checkAddrWithEnv` / `proveAddrWithEnv` on the full-closure path. `--subject-only` still uses Lean `buildVerifyConst` + `executeIxVM` / `proveIxVM` (witnesses intentionally small there). # New crate `crates/ix/src/env_handle.rs` — the `EnvHandle` struct + constructors live in the existing `ix` crate next to the witness builder and codegen runner. # Sanity * Multi-target check (warm): `ix check --ixe init.ixe Nat.add_comm Nat.add` → 3.3 s wall, single envHandle shared across both targets. * Shard 26 check (warm): `ix check --ixe init.ixe --ixes init.ixes --shard 26` → ~78 s wall, FFT = 107_006_963_281 (parity preserved). # Coverage | Command | Path | |--------------------------------------|-----------------------------------| | `ix check NAME --ixe` | `checkAddrWithEnv` | | `ix check NAME` (no `--ixe`) | `checkAddrWithEnv` + fromBytes | | `ix check --claim hex --ixe` | `checkAddrWithEnv` for check-none | | `ix check --ixes --shard K` | `shardCheckWithEnv` | | `ix check --ixes` | `shardCheckWithEnv` × all shards | | `ix prove NAME --ixe` | `proveAddrWithEnv` | | `ix prove --ixes --shard K` | `shardProveWithEnv` | | `ix prove --ixes` | `shardProveWithEnv` × all shards | | `Benchmarks/Typecheck` Phase 1+2 | `checkAddrWithEnv` / `proveAddrWithEnv` | Only `--interp` and `--claim hex` over a non-`check addr none` persisted claim still build a Lean `ClaimWitness`; both go via the `.leanW` target arm. * ix codegen: add --check flag for CI; gate stale generated kernel `ix codegen --check` compares the emitted Rust source against the on-disk `crates/ix/src/aiur_ixvm.rs` and exits 0 if identical, 1 otherwise. No write side effect. ~2 s warm; fast enough to gate on every PR. Wired into the `lean-test` CI job so a forgotten regen on a kernel-touching PR fails CI instead of merging stale generated code that drifts from the Bytecode → Rust emitter. * ix check: --interp {source|bytecode} to bypass codegen kernel Two interpreter modes behind a single `--interp` flag: * `--interp source`: Aiur source interpreter (`Aiur.runFunction` over the source-level `Decls`). Richer per-step error diagnostics. * `--interp bytecode`: generic Aiur bytecode interpreter (`Bytecode.Toplevel.execute`, `rs_aiur_toplevel_execute` route). Skips the `ix codegen` + `cargo build --release` cycle needed after editing `Ix/IxVM/*.lean` — the bytecode is rebuilt Lean-side at exe load. Slower per-check than the codegen kernel; ideal for tight iteration on the IxVM source. * omit the flag entirely for the native codegen kernel (default). Invalid values (e.g. `--interp foo`) fail fast with a clear error. # Plumbing The `check_addr_with_env` and `shard_check_with_env` FFIs take a `use_bytecode: bool` and dispatch via a shared `dispatch_execute` helper. `--interp bytecode` sets it to `true`. The `.leanW` arm of `runCompiled` also picks between `Bytecode.Toplevel.execute` and `executeIxVM` based on the same flag. The `--interp source` path requires a Lean `ClaimWitness`. The existing driver had switched to `.addr`/`.shard` targets against the Rust-owned `EnvHandle`; source-interp had no way to materialise those. `forEachClaim` now takes a `forceLeanWitness : Bool` — when true, it builds a Lean witness for every target (via `mkWitness` / `loadIxonEnv` for the compiled-Lean-env path) and passes `.leanW` so `runInterp` can consume it directly. # Sanity * `ix check Std.Time.Week.Offset.ofMilliseconds` (warm): 9.3 s wall, FFT = 12_430_516_949 (codegen kernel). * `ix check --interp bytecode Std.Time.Week.Offset.ofMilliseconds` (warm): 14.6 s wall, FFT = 12_430_516_949 (bytecode interp). * `ix check --interp source Eq` (warm): 6.3 s wall, output `Eq: ()`. * `ix check --interp garbage Nat.add_comm`: exits 1 with clear error. * IxVM codegen: regen + rebase-fixups after KLevel port Aligns the ap/codegen-ixvm-native stack with main's KLevel pointer port (b84a500) and re-lands clippy-clean under the workspace lint set: - Regen crates/ix/src/aiur_ixvm.rs from Aiur source (KLevel = &KLevelNode changes the generated fn shapes; ~26k lines net churn). - Extend the generated file's #![allow(...)] header (via Ix/Aiur/Stages/Codegen.lean) with clippy::ptr_as_ptr, clippy::match_same_arms, and clippy::large_types_passed_by_value — three lints the codegen's straight-line style trips deterministically on every regen and that clippy::all doesn't cover. - rustfmt reflow in crates/aiur/src/execute.rs and a handful of ix / ffi files that CI now enforces via the ap/aiur-aux-recursor-parity toolchain. - Fix 4 real clippy warnings uncovered under the workspace lint set: * env_handle.rs: clone-on-Copy → deref (ReducibilityHints is Copy). * aiur_ixvm_witness.rs: collapse if-let-if, drop needless continue, slice::from_ref instead of &[x.clone()]. * aiur_ixvm_runner.rs: keep execute_ixvm's Vec<G> arg (required by AiurSystem::prove_ixvm's fn-pointer bound) + local #[allow( clippy::needless_pass_by_value)] with a comment explaining why. * ffi/aiur/protocol.rs: unqualify aiur::G/IOBuffer/QueryRecord after the workspace lint added unused_qualifications; is_multiple_of() over % 0. Verified: cargo clippy --workspace --all-targets --all-features --D warnings clean; lake test -- --ignored ixvm passes. * ix-check + ixvm tests: coverage gate, --interp source fix, codegen parity + rename ix crate to ixvm-codegen Four review fixes on the codegen-ixvm-native stack. - **Coverage gate on \`--ixes\` (all-shards, native path).** \`runShardManifestAllNative\` now calls \`shardsCover\` before running any shard, matching the pre-refactor \`runShardCheckAll\` behavior. Without this, a stale/truncated \`.ixes\` manifest makes \`ix check --ixes\` exit 0 while some env constants are never checked by any shard — the mundane trigger being: edit a definition, rebuild the \`.ixe\`, forget to re-run \`ix shard\`, and the constants that go unchecked are exactly the ones just edited. Single-shard \`--shard K\` path is unchanged (consistent with pre-refactor). \`shardsCover\` moved above \`runShardManifestAllNative\` to satisfy forward-decl ordering. - **\`ix check --ixe … --claim <hex> --interp source\` regression.** The \`claimHex\` arm in \`forEachClaim\` ignored \`forceLeanWitness\` and always mapped \`.check addr none\` claims to \`Target.addr\`. Under \`--interp source\`, \`runInterp\` then rejects \`.addr\` with "\`--interp requires a Lean witness; addr/shard targets unreachable here\`". Now mirrors the sibling name-based arms and routes through the Lean witness path when \`forceLeanWitness\` is on. Only bit the common claim shape. - **Codegen ↔ bytecode parity CI test.** New \`runParityCase\` + \`parityCases\` in \`Tests/Ix/IxVM.lean\` wire every constant already listed in \`kernelCheckEntries\` plus \`kernel_unit_tests\` through BOTH \`Toplevel.execute\` (bytecode) and \`Toplevel.executeIxVM\` (codegen'd Rust), and diff the returned \`(output, IOBuffer, QueryCounts)\` triples. Turns "generated kernel ≡ interpreter on QueryRecord" from reviewed-by-hand into checked-by-CI. 129 assertions run per suite invocation (43 constants × output + IOBuffer + QueryCount), all pass on this branch head. - **Rename \`crates/ix\` → \`crates/ixvm-codegen\`.** The old \`ix\` crate held only the codegen'd IxVM kernel + its runner + witness helpers. \`ix-kernel\` is the hand-written Rust ix typechecker; \`ix-common\` / \`ix-compile\` are Ix-level infrastructure. \`ixvm-codegen\` cleanly names what this crate holds. Path updates: workspace \`members\` and \`workspace.dependencies\`, ffi crate dep + \`use\` sites, Lean \`codegenOutPath\` in \`Ix/Cli/CodegenCmd.lean\`, doc reference in \`Ix/Aiur/Semantics/BytecodeFfi.lean\`. \`ix codegen\` now writes to \`crates/ixvm-codegen/src/aiur_ixvm.rs\`. Verified: \`cargo check --workspace\` + \`lake test -- --ignored ixvm\` both exit 0 post-rename; parity + pinned FFT tests all pass. * cargo-deny: allow ar_archive_writer's Apache-2.0 WITH LLVM-exception Post-rename, ixvm-codegen pulls stacker → psm → ar_archive_writer (a build-time transitive). ar_archive_writer is licensed under the LLVM-exception variant of Apache-2.0 (standard for LLVM tooling; no additional runtime obligation for downstream users), which isn't on the workspace allow list. Add a per-crate exception scoped to ar_archive_writer so we don't blanket-allow the license family across the tree.

arthurpaulino · 2025-02-04T13:52:08Z

No description provided.

Lays the groundwork for measuring the kernel performance plan (plans/okay-let-s-write-a-lucky-dolphin.md) per audit §10. Counters live in a new kernel::perf module; KEnv carries a PerfCounters field that is dumped from a Drop impl when IX_PERF_COUNTERS=1 is set. Unset is the production default — every increment short-circuits via a LazyLock<bool> so the cost is a single cached branch on the hot path. Wired into the cache get sites the audit identified: - whnf_cache hits/misses (whnf.rs around 201/211) - whnf_no_delta_cache hits/misses (whnf.rs around 437/446) - infer_cache and infer_only_cache hits/misses (infer.rs around 45/51) - def_eq_cache hits/misses (def_eq.rs around 137/149) - def_eq_failure set hits/inserts (def_eq.rs around 360) - per-constant peak/avg MAX_REC_FUEL consumption (recorded in TypeChecker::reset before the next constant resets it) Verified against lake test -- kernel-check-env --ignored: 192156/192156 constants pass in 251s (4% over the 242s baseline; overhead is the LazyLock branch + atomic load in the disabled path). P1: def_rank_id replaces def_weight_id for hint-priority comparison Audit Tier 1 #3 (kernel-perf-adversarial-audit-2026-04-26.md, §4.2): the prior u32 encoding mapped Abbrev to u32::MAX-1 and saturating-added Regular(h) to h+1, which collide at h ≥ u32::MAX-2. When that happens the delta-direction logic treats Abbrev and a maximally-heavy Regular as "same height" and unfolds both, instead of preferring Abbrev as Lean does (compare(d_t->get_hints(), d_s->get_hints()) at type_checker.cpp:910). Replace the u32 weight with a (class: u8, height: u32) tuple compared lexicographically: - Opaque / Theorem / unknown → (0, 0) - Regular(h) → (1, h) (height ordering preserved within class) - Abbrev → (2, 0) (strictly above every Regular) Update the two call sites (is_def_eq lazy-delta height comparison and lazy_delta_step). The map_or default for "missing head" is preserved as (u8::MAX, u32::MAX) — the branch is dead in practice (a_delta && b_delta imply both heads are present) but kept consistent with the prior u32::MAX sentinel. Two regression tests: - def_rank_abbrev_above_saturated_regular: Abbrev outranks Regular(u32::MAX) (the previous saturation collision). - def_rank_regular_orders_by_height: height monotonically orders Regular ranks within the class. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 256s (no regression vs the 242s pre-perf-counters baseline; the +14s is from the IX_PERF_COUNTERS=unset LazyLock branch added in the prior commit, not this change). P2: peel_proj_forall fast-paths syntactic Pi in projection inference Audit Tier 1 #2 (kernel-perf-adversarial-audit-2026-04-26.md, §7.2): infer_proj's two parameter-consuming loops (param peel and field peel) called self.whnf(&r)? unconditionally per iteration, on a body mutated by subst at the previous step. The whnf cache rarely hits between iterations and each call re-traverses the substituted body. Extract a peel_proj_forall(&r, err) helper that: - tries ExprData::All(..) syntactically first (no WHNF call), and - falls back to full self.whnf(e) only when the binder isn't already syntactic Pi. This mirrors Lean's inferProj at type_checker.cpp:582–610. Both projection-inference loops now call peel_proj_forall instead of unconditional whnf. Behaviorally equivalent — same WHNF semantics on miss, no semantic change otherwise. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 258s (parity with the post-pre-work, post-P1 baseline; no measurable regression and the cache-hit-rate counters will move on tactic-heavy workloads under IX_PERF_COUNTERS=1). P3a: WhnfFlags substrate (no behavior change) Lays the foundation for the Lean4Lean architectural alignment described in plans/okay-let-s-write-a-lucky-dolphin.md. Phase 3a is substrate-only — no call site is migrated to cheap mode yet, so behavior is unchanged. Adds: - WhnfFlags { cheap_rec, cheap_proj } with FULL and CHEAP consts and is_full(). CHEAP is currently equal to FULL until Phase 3c wires it. - whnf_core_with_flags (private): the existing whnf_core impl, now threading flags into recursive calls and try_iota_with_flags. - whnf_core / whnf_core_cheap (super): FULL/CHEAP wrappers. - whnf_no_delta_with_flags (private): the existing whnf_no_delta impl with the Prj branch gated on cheap_proj — falls back to full whnf on the projected value when not cheap. - whnf_no_delta (pub) / whnf_no_delta_cheap (super): wrappers. - try_iota_with_flags: gates major-premise WHNF and string-literal constructor reduction on cheap_rec. - try_proj_app_reduce_with_flags: gates projected-value WHNF on cheap_proj. Cache reads/writes (whnf_no_delta_cache, equiv-manager second-chance) are gated on flags.is_full(): cheap callers neither read nor write the cache, preserving the invariant that any cached entry is a fully-reduced normal form. Phase 3b will inline the projection branch into whnf_core to match Lean4Lean's two-layer architecture (refs/lean4lean/Lean4Lean/ TypeChecker.lean:266, 297). Phase 3c will flip CHEAP to enable cheap_proj and migrate specific def-eq sites. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 243s (matches the 242s baseline; substrate adds no measurable overhead when CHEAP == FULL). P3b: inline projection into whnf_core (Lean4Lean architectural alignment) Move the Prj branch from whnf_no_delta_with_flags into whnf_core_with_flags so our whnf_core matches Lean4Lean's whnfCore semantics exactly (refs/lean4lean/Lean4Lean/TypeChecker.lean:284-292, 337-341). Before this commit: whnf_core — beta + zeta + iota + cheap projection (recursive whnf_core on val, no delta) whnf_no_delta — whnf_core + FULL projection (full whnf on val) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta Lean4Lean's architecture has no whnf_no_delta layer. Their whnfCore includes projection, with the cheap_proj flag deciding whether the projected value uses whnfCore (cheap) or whnf (full). After this commit: whnf_core (with WhnfFlags) — beta + zeta + iota + projection (cheap_proj controls val reduction) whnf_no_delta — whnf_core(_, FULL) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta The bare-Prj branch in whnf_no_delta_with_flags is removed — whnf_core now handles it directly. The App-of-Prj branch stays in whnf_no_delta because whnf_core's loop returns once the outermost Prj is resolved; try_proj_app_reduce_with_flags gives one more attempt at the same cheap_proj policy when the outer expression is App(Prj, ...). Pure refactor, no semantic change with CHEAP == FULL. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 270s (within noise of the 243s pre-refactor baseline). Phase 3c will flip CHEAP to enable cheap_proj=true and migrate def-eq's lazy-delta sites surgically per Lean4Lean's pattern. P3c (postponed): document the HeaderParsedSnapshot regression P3a (substrate) and P3b (Lean4Lean architectural alignment) are committed. Phase 3c — flipping CHEAP to enable cheap_proj=true and migrating the def-eq lazy-delta sites — was attempted but reproduced 5 failures on chained projections in Lean.Language.Lean.HeaderParsedSnapshot.* even after P3b inlined the projection branch into whnf_core. The substrate is left in place; CHEAP stays equal to FULL until the regression's root cause is understood. Notes on the regression for the next investigator: - Failures: HeaderParsedSnapshot.{stx,result?,metaSnap,toSnapshot,ictx}, all with 'projection: type mismatch with declared struct'. - The struct `extends` a parent, so each projection is a chained Prj whose val is itself a Prj into the parent. - The error comes from infer.rs's infer_proj at the head-vs-struct_id address compare, after FULL whnf on val_ty. That whnf is FULL, but val_ty was inferred via paths that may have consulted a def-eq cache populated under cheap mode. Possible cache-poisoning suspect: def_eq_cache writes a `false` result for inputs whose lazy-delta loop bottomed out under cheap projections. The cache key uses raw a/b hashes, not cheap-reduced shapes, so a stored `false` is indistinguishable from a FULL `false` by future readers. - Lean4Lean does not have an analogous wide def_eq_cache; their failure cache is keyed only on same-spine pairs in lazyDeltaReductionStep. Future P3c iterations should either prove the cache poisoning theory incorrect or restrict def_eq_cache writes to FULL-derived results.

* Refactor Claim ADT + add merkle infrastructure for ZK aggregation Adds the format hooks needed before recursive verification lands in Aiur. Five-variant Claim ADT with explicit assumption commitments, a canonical Blake3 merkle module, a serializable AssumptionTree for recovering leaf sets, and a Contains claim for the discharge step. Claim ADT (5 variants): - Eval { input, output, assumptions: Option<Address> } - Check { const_addr, assumptions: Option<Address> } - CheckEnv { root, assumptions: Option<Address> } - Reveal { comm, info } -- unchanged, no assumptions - Contains { tree, const_addr } -- new, for inclusion proofs Tag4 reorganized to keep everything in single-byte tags: - 0xE for env, comm, AssumptionTree, and claims (slots 0-7) - 0xF for proofs (slots 0-4; 5-7 reserved) - Comm moved from variant 5 -> 1 Matches the "Variant (0-7)" constraint documented in docs/Ixon.md. Env serialization: - Every .ixe file now carries a canonical merkle root over its consts.keys() in the on-disk header (non-optional, 32 bytes; empty const sets use the zero-address sentinel). - Two envs with the same const set produce byte-identical roots regardless of insertion order. Verified on deserialize. New modules: - src/ix/ixon/merkle.rs + Ix/Merkle.lean: canonical sorted builder, free-form merkle_join composition, membership proofs, domain separation per RFC 6962. - src/ix/ixon/assumption_tree.rs + Ix/AssumptionTree.lean: serializable merkle tree with Leaf/Padding/Node variants. canonical() builds the same shape merkle_root_canonical hashes; join() is O(1) free-form composition. - src/ix/kernel/claim.rs: builders that compute transitive-dep assumptions from an env (build_check_claim, build_eval_claim, build_check_env_claim, env_merkle_root). Other: - Extracted shared BFS walker on Env::bfs_refs + Env::transitive_deps_excl; the inlined test-feature copy in lean_env.rs now calls into it. - Lean FFI export rs_env_merkle_root for cross-impl verification of the env root. - Proof bytes for all variants are uniform opaque ZK bytes; witness data (e.g., Contains merkle paths) is prover-side scratch consumed by the ZK circuit and not transmitted on the wire. - docs/Ixon.md Tag4 tables and env section updated; .ixe extension documented. Recursive verification (the ZK proof generation for Contains and the aggregation discharge transitions) is intentionally deferred to a follow-up. Tests: 993 Rust unit tests pass (was 953 pre-refactor), 813 Lean tests pass with no failures; cargo clippy clean. * Add lazy env deserialization and anon-mode kernel FFI Two related changes to reduce kernel memory + lay groundwork for metadata-isolated typechecking. **Lazy constant deserialization.** `Env::consts` now stores `LazyConstant` (`Arc<[u8]>` + `OnceLock<Arc<Constant>>`) instead of `Constant`. Constants are materialized on first access and the structured form is cached. Sparse access patterns (single-constant typecheck, transitive_deps walk, claim builders) only parse the closure they need; non-reachable constants stay as raw bytes. The `.ixe` section-2 layout gains a Tag0 length sidecar before each constant's Tag4 bytes: old: [addr:32] [Tag4 constant bytes] new: [addr:32] [Tag0 length] [Tag4 constant bytes] The length is section-level framing, not part of the constant's content hash. `Address::hash(raw_bytes) == addr` is preserved. `Constant::put`/`Constant::get` are unchanged. The Lean side (`Ix.Ixon.putEnv`/`getEnv`) reads/writes the new format. **Anon-mode kernel FFI.** New `rs_kernel_check_consts_anon(path, addrs, quiet)` exposes anonymous-mode typechecking by content address. The kernel runs as `KEnv<Anon>` / `TypeChecker<Anon>` with every `M::MField<T>` erased to `()`, so the typechecking logic structurally cannot read metadata. Useful for zkPCC verifiers that hold only addresses. Supporting pieces: - `KernelMode::HAS_META: bool` for future compile-time gating. - `AnonEnv<'a>` wrapper (`src/ix/kernel/anon_env.rs`) exposing only consts/blobs/transitive walks — no `named`/`names`/`comms`. - `Env::get_anon` reads header + blobs + consts, parse-and-drops metadata sections (3-5), returns an `Env` with empty `named`/ `names`/`comms`. Same merkle-root verification as `Env::get`. - `rs_de_env_anon` FFI + `Ix.Ixon.rsDeEnvAnon` Lean wrapper. - `Ix.KernelCheck.rsCheckConstsAnonFFI` Lean binding. Caveat: `ixon_ingress::<Anon>` still consults `Env::named` internally to enumerate work items. The resulting `KEnv<Anon>` is metadata-free so the typechecker is anon, but full ingress-level metadata isolation is a follow-up. Tests: 16 new (9 lazy + sparsity; 3 AnonEnv; 4 get_anon). All 1009 Rust unit tests pass; all 813 Lean tests pass; clippy clean. * Simplify ix check + add metadata-free anon mode - `ix check` becomes .ixe-only with a positional `<path>`. Drops the `--lean` compile-and-check flow and the `--env` flag; direct Lean → kernel checks remain available through `rsCheckConstsFFI` for tests. Removes the redundant `CheckIxonCmd.lean`. - New `--anon` flag runs a metadata-free kernel check: loads the .ixe via `Env::get_anon` (discards named/names/comms), enumerates work items from `env.consts` alone, and rebuilds member + ctor projection addresses deterministically via `Constant::commit`. Rejects `--consts`/`--ns`/`--consts-file` since it always checks everything. - Compiler: unwrap singleton non-inductive Muts blocks to standalone Defn/Recr. `Expr::Rec(0, univs)` already resolves correctly against a one-member block, so the wrapper was pure overhead; this keeps the env structurally uniform and matches `compile_single_def`. The anon ingress for standalones passes `mut_ctx_override = [self_id]` so self-recursive standalones still typecheck. - Kernel: anon parallel runner mirrors `run_checks_parallel_on_large _stacks` (32 workers, slow-detection, RSS tracking). Genericized `check_one_const<M>` / `check_consts_loop<M>` / `format_tc_error<M>` to share the runner. `KernelMode::meta_field_with` / `meta_field_try` gate metadata lookups in expression ingress at compile time. - Anon lazy ingress (`LazyAnonIngress` in tc.rs, helpers in ingress.rs) dedupes blocks via `kenv.blocks.contains_key(&KId<Anon>(B, ()))` before re-running `ingress_anon_block` (prevents N² re-ingestion when sibling projections fault separately within one worker check). - `Env::get_anon` now harvests `ReducibilityHints` from the otherwise-discarded Named section into a sidecar `Env::anon_hints: FxHashMap<Address, ReducibilityHints>`. Anon ingress threads these through a new `hints_override` parameter on `ingress_defn` so the lazy-delta tiebreak in `def_eq::def_rank_id` sees realistic heights instead of `Regular(0)`. Hints are performance advice, not correctness data — supplying them in anon mode preserves the metadata-free trust model. - Regenerate the canonical addresses in `PrimAddrs::new()`; singleton- unwrap changed the content hash of every self-recursive primitive (Nat.add, String, Nat.rec, …). End-to-end on compileinitstd.ixe (105,487 constants): meta: 105487/105487 in 20.6s, peak RSS 7.4GB anon: 89010/89010 in 17.8s, peak RSS 4.0GB * Clean up cargo clippy --all-targets - 11× `cloned_ref_to_slice_refs`: `&[x.clone()]` → `std::slice::from_ref(&x)` in `assumption_tree.rs` + `merkle.rs` test modules. Same allocator footprint, no clone, matches the lint's recommended idiom. - 2× `useless_vec`: `vec![0xE3, 0x00, 0x00]` → `[0xE3, 0x00, 0x00]` in serde-reject tests where the buffer is only borrowed. All 1011 unit tests pass. `cargo clippy --all-targets` is now clean. * ix check: print full content hashes + add --workers flag - The anon-mode progress / failure-log labels and the meta-mode hash-display fallback were truncating addresses to 16 hex chars (`@1f4b195aefa10e26`). Drop the `[..16]` slice so the full 64-char Blake3 hex is printed (`@1f4b195aefa10e2690d13c5b98b3d9124d3fbb5c…`), matching what tooling like `--fail-out` records and what the metadata-free workflow actually needs to identify a constant. - `--workers N` flag on `ix check` plumbs through the existing `IX_KERNEL_CHECK_WORKERS` env var that `resolve_kernel_check_workers` in `src/ffi/kernel.rs` reads. Useful for isolating per-worker memory cost (`--workers 1` lets you see the env's static footprint without per-worker overhead) and for capping concurrency in resource-tight contexts. * Mmap .ixe + cache-free LazyConstant for anon mode `lake exe ix check compilemathlib.ixe --anon` on a 3.2 GB env now peaks at ~11 GB RSS, down from ~40 GB; build_anon_work is 50× faster. Three intertwined changes: 1. Memory-map the .ixe (avoids ~3.2 GB heap copy of bytes) - Cargo: add memmap2 = "0.9". - `BytesSource` enum in `src/ix/ixon/lazy.rs` distinguishes heap-resident `Arc<[u8]>` from `(Arc<Mmap>, offset, len)` windows into a memory-mapped file. `LazyConstant::raw_bytes`, `verify_address`, `PartialEq` route through `BytesSource::as_slice` so every existing consumer is mmap-transparent. - `Env::get_anon_mmap(path)` in `src/ix/ixon/serialize.rs` opens the file, mmaps it, and stores Section-2 consts as mmap-backed `LazyConstant`s. Sections 3-4 (names + named) are still parsed transiently to harvest hints, then dropped before return. - `rs_kernel_check_anon` (`src/ffi/kernel.rs`) switches to `get_anon_mmap`; the old `std::fs::read` + `Env::get_anon` heap path is gone for the anon FFI. 2. Strip the persistent `LazyConstant` parsed-Constant cache - `LazyConstant.cache` was `Arc<OnceLock<Arc<Constant>>>` and accumulated forever — for mathlib that meant ~30 GB of parsed `Arc<Expr>` trees pinned in the env across the entire run. - New shape: `cache: Option<Arc<Constant>>`. Populated only by `from_constant` (compile-side, where we already own the parsed value). `from_bytes`/`from_mmap_slice` set `None`, and their `get()` parses fresh on every call without storing. - Re-parse cost is bounded by the existing per-worker dedup: `kenv.consts` already addresses each ingressed constant once per work item, and `clear_releasing_memory()` drops the kenv between items. The kenv is now the only persistent materialization layer. 3. `LazyConstant::peek_variant` for `build_anon_work` - One-byte read of the outer Tag4 head to identify the `ConstantInfo` variant — no body parse, no allocation. - `build_anon_work` now dispatches on `peek_variant()`; only `Muts` blocks trigger `lc.get()` (we need the member list for projection-address enumeration), and that `Arc<Constant>` drops at the end of the match arm. - Previously every constant was fully materialized at startup just to read its variant tag. With ~95% of the env being standalone/projection, the work-enumeration pass now skims the env at near-IO speed. Test updates: - `mmap_slice_roundtrips` (already added with the earlier mmap scaffolding) exercises the mmap window roundtrip. - New `from_bytes_does_not_cache` and `from_constant_clones_share_cache` document the new caching contract. - `peek_variant_*` tests cover every variant + empty-bytes and unknown-flag error paths. - `lazy_sparsity_only_materializes_closure` in `src/ix/ixon/env.rs` reframed to assert BFS-of-closure correctness instead of cache side-effects (`is_materialized()` no longer fires for lazy loads). End-to-end on compilemathlib.ixe (--anon, 32 workers): before: 640658/640658 in 266s, peak RSS ~40 GB after: 640658/640658 in 223s, peak RSS 11.3 GB * Address review findings: correctness + small refactors Code-review pass over the anon-mode / mmap / cache-strip work. This commit lands the highest-value subset — the correctness fixes and the small refactors that prevent future drift — and leaves the larger items (format-version bump, sealed marker trait, AnonEnv audit) for separate follow-ups. Correctness: - Verify per-constant address on load (#1). `LazyConstant::verify_address` existed but was never invoked from `Env::get`, `get_anon`, or `get_anon_mmap`. The env-level merkle root catches missing/extra entries but not byte-tampering of a constant whose key is intact; without this check, corruption surfaced much later inside `LazyConstant::get` with a misleading parse error. Inline the `Address::hash(bytes) == addr` check in each loader's Section-2 loop. Added 3 corruption-detection tests (`env_const_bytes_tampering_*`). This required updating a handful of existing tests that stored constants under fake `Address::hash(b"a")` keys instead of their true content hashes — round-tripping such envs now correctly rejects. Added a `store_canonical(env, c) -> Address` test helper for the canonical pattern, and `*_discriminator(refs, n)` so tests can produce content-distinct constants when the same ref-set would otherwise collide. - Hard-error in `ixon_env_to_decoded` on parse failure (#6). `src/ffi/ixon/env.rs` used `filter_map` to silently drop any const whose bytes failed `LazyConstant::get` — the Lean caller had no signal of the lost entries. Switch to a Result-collecting loop and propagate the first parse error; update both FFI call sites (`rs_de_env`, `rs_de_env_anon`). - Doc fixes (#4, #5). `docs/Ixon.md`'s AssumptionTree section described only `Leaf=0x00` and `Node=0x01`, missing `Padding=0x01` (and the docs' `Node` tag collided with the real `Padding` tag — the real `Node` is `0x02`). Also: the env-section table cell said "opt-tagged merkle root" but the implementation writes a bare 32-byte address. Refactors that prevent future drift: - Canonical projection-address helpers (#9). Four production sites reconstructed `Constant::new(IxonCI::{D,I,R,C}Prj{...}).commit().0` by hand; the anon pipeline silently breaks if any one drifts. Extract `defn_proj_address` / `indc_proj_address` / `recr_proj_address` / `ctor_proj_address` (plus `_constant` variants) in `src/ix/ixon/constant.rs`. Update `compile.rs`, `compile/mutual.rs`, and the anon `*_proj_addr` helpers to call them. - `verify_proj_addr_in_env` helper (#21). `ingress_anon_block` had the same "computed-address not in env" check repeated four times (DPrj/RPrj/IPrj/CPrj). DRY into one helper that produces a consistent error format. UX: - Anon display labels switched to `#<hex>` everywhere (#18). Rust was emitting `@<hex>` in progress/fail-out; Lean's `runCheckAnon` was emitting `#{i}` (result index). Standardize on `#<hex>` so the CLI failure summary is joinable with the fail-out file. This required exposing addresses per result slot to Lean — `rsCheckAnonFFI` now returns `Array (String × Option CheckError)` pairing each result with its content-address hex string. Rust builds the pairs via a new `build_anon_result_array` helper. - Stale "check-ixon" header in FailureLog (#16). One-line rename in the doc comment + `# ix check-ixon failures` writeln to `# ix check failures`. Tests: - `testPrimitivesParity` (PrimAddrs regen-parity test). Catches silent drift between hardcoded primitive addresses in `PrimAddrs::new()` and what `rsCompileEnvFFI` produces from the live Lean primitives. Plumbing: new `PrimAddrs::lean_parity_table()` returns `(lean_name, hex)` pairs; new `rs_prim_addrs_canonical` FFI exposes them to Lean; new test in `BuildPrimitives.lean` iterates `kernelPrimitives`, compares against the hardcoded table, and fails with a printable diff on mismatch (with instructions to regenerate). Skipped: `eagerReduce` — synthetic kernel marker whose PrimAddrs value (`0xff…3`) intentionally diverges from its compiled content hash (which collides with `id`). - 3 new corruption-detection tests in `serialize.rs` exercise the Section-2 verify check for `Env::get`, `Env::get_anon`, `Env::get_anon_mmap`. Verification: `cargo test --lib` 1025 passing (3 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe` 105487/105487 in ~21s, `--anon` 89010/89010 with peak RSS 1.4 GB, `.lake/build/bin/IxTests --ignored rust-kernel-build-primitives` shows both `build primitives dump` and `primitive address parity (PrimAddrs vs live compile)` pass. Out of scope (deferred to follow-ups): - #2 Format version bump (Env::FLAG) - #3 rs_kernel_check_consts_anon still uses Env::get on main - #8 Sealed marker trait for the lazy_anon transmute - #11/#12 AnonEnv audit (vestigial wrapper) - #13, #15, #17, #19, #20, #22-25 Various quality cleanups * Round-2 review fixes: mmap defense + small cleanup Six small items from the round-2 review of the anon-mode work: - N1: stat-at-open defense in `Env::get_anon_mmap` (`src/ix/ixon/serialize.rs`). Capture the file size before mmap; if the kernel's mapped length disagrees (file truncated between open and map) bail with a clear error instead of letting workers SIGBUS deep in `LazyConstant::get`. Truncate-in-place under a live mapping is still undefined per POSIX — documented as a caller contract — but the open-time check catches the common case. - New `get_anon_mmap_survives_file_unlink` test: load via mmap, unlink the path, then materialize both already-touched and not-yet-touched constants. Locks in the inode-retention invariant that the SIGBUS analysis depends on; a future refactor that switched to `mmap_anonymous` or copied bytes into a tmpfile would fail loudly here instead of letting workers SIGBUS in production. - #11: delete `AnonEnv::as_ixon_env_unchecked` (`src/ix/kernel/anon_env.rs`). Dead `pub(crate)` escape hatch marked `#[allow(dead_code)]` — confirmed zero callers and removed. - #13: `debug_assert!` on `OnceLock::set` for both result vectors (`src/ffi/kernel.rs`, meta + anon paths). The previous `let _ = results[idx].set(...)` silently dropped a re-set. If a future `build_*_work` dedup refactor breaks the one-write-per-slot invariant, debug builds now panic with the slot index instead of silently losing results. - #19: `allNames.contains` O(n²) preflight → `Std.HashSet` lookup (`Ix/Cli/CheckCmd.lean`). At mathlib scale (~700k env names × thousands of seed names) the previous linear scan-per-name spent measurable seconds on missing-name preflight alone. - N4: doc imprecision in `src/ix/ixon/lazy.rs` — the cache-policy preamble said the worker `KEnv` is "cleared between work items" but the actual cadence is `clear_every` items (`IX_KERNEL_CHECK_CLEAR_EVERY`, default 1 but tunable). Refined wording so the doc matches the code. Verification: `cargo test --lib` 1026 passing (1 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe --anon` 89010/89010 in 15.2s peak RSS 1.4 GB (regression-clean). Out of scope, separate follow-ups: - #2 Format version bump - #3 rs_kernel_check_consts_anon zkPCC FFI (Env::get → mmap) - #8 Sealed marker trait for the lazy_anon transmute - #10 child_arena closure side-effect (widen MField to tuple) - #12 AnonEnv duplicates bfs_refs / transitive_deps_excl - #14 anon worker bypasses M-generic display helpers - #17 Anon --fail-out docstring claims --consts-file compat - #20 Singleton-unwrap duplicated across compile paths - #22-24 Lean `partial def`, `.get!` in tests, unguarded as-u64 casts - N2/N3 Lean is_materialized() callers + compute_const_size_breakdown - Test gaps: rs_kernel_check_anon integration, concurrent mmap, verify_address-on-mmap-corruption, etc. * rustfmt * Tests: derive RawConst addrs from content (verify_address fix) The `Address::hash(bytes) == addr` defense added in f792102 rejects `RawConst { addr, const }` pairs where `addr` is uncorrelated with `serConstant const`. The Rust-side tests in `serialize.rs` were updated at the time (`store_canonical` helper), but three Lean-side sites still produced mismatched pairs and surface now as: × ∃: Env serialization Lean==Rust × ∃: serde RawEnv with data × ∃: serde RawEnv roundtrip deserialization failed: rs_de_env: Env::get: const at idx 0 bytes hash to 6110b739… but stored under b177ec1b… Three fixes, all derive `addr := Address.blake3 (serConstant c)`: - `Tests/Gen/Ixon.lean::genRawConst` — property-test generator was `RawConst.mk <$> genAddress <*> genConstant` (uncorrelated). Now generates the const first, then derives addr from `serConstant`. - `Tests/FFI/Lifecycle.lean::serdeTests` (`withData` unit case) — was using one `testAddr := Address.blake3 #[1,2,3]` for both the const and unrelated blob/comm slots. Split into `testAddr` (still used for the content-hash-free blob/comm) and `testConstAddr := Address.blake3 (serConstant testConst)`. Added a name entry for the new canonical addr. - `Tests/FFI/Lifecycle.lean::genSerdeRawEnv` — the "pool of addresses" pattern picked const addrs from a random pool that couldn't possibly match content hashes. Restructured: consts derive canonical addrs from content, each gets its own name-table entry appended to the pool-derived entries so the serde pipeline's "all addresses resolvable" invariant still holds. * Regenerate Aiur primitive addresses for singleton-unwrap Compiler's singleton non-inductive Muts unwrap (12630aa) changed the content hashes of 48 primitives (`Nat.rec`, `Nat.add`, `Nat.mul`, …). The Aiur kernel's `Ix/IxVM/Kernel/Primitive.lean` still held the pre-unwrap blake3 bytes, so primitive dispatch missed every affected constant and fell through to structural `Nat.rec` unfolding. Tests like `IxVMPrim.nat_mul_big` then drove millions of Nat.succ unfoldings inside the Aiur trace, OOMing the prover. Addresses regenerated from `PrimAddrs::lean_parity_table()` (which the branch had already updated in `src/ix/kernel/primitive.rs`). * Fix standalone Recr ingress: typ-based inductive lookup `find_matching_block_addr` heuristic in the standalone-Recr branch of `build_convert_inputs_walk` picks the inductive block by matching ctor count to rule count. When multiple in-scope inductives share the same ctor count, it returns the wrong block — the stored rules' `ctor_idx` then points to a sibling's ctor, while `populate_rules` (canonical) derives `ctor_idx` from the right inductive's `ctor_indices`. The mismatch surfaces as `compare_rules`'s `assert_eq!(s_ctor, c_ctor)` panic (e.g. `38 != 40` for `IxVMPrim.nat_land_lit`). Switch the standalone-Recr path to the same typ-based resolution `build_aux_recr_ctor_idxs` already uses for aux Recr blocks: peel `params + motives + minors + indices` foralls of `recr.typ`, take the major's head Ref, look it up in `refs` to get the inductive's address, then resolve `IPrj.block` for the Muts wrapper. Slice ctor positions for the specific member via `extract_member_ctor_idxs`. Also store the resolved Muts `block_addr` in `CKRecr` (was passing the heuristic's wrong address through to `convert_recursor`). --------- Co-authored-by: Arthur Paulino <arthurleonardo.ap@gmail.com> Co-authored-by: Samuel Burnham <45365069+samuelburnham@users.noreply.github.com>

plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed).

* Aiur Bytecode → Rust codegen for the IxVM kernel Adds a new Aiur pipeline stage that translates `Bytecode.Toplevel` into a Rust source module, one `fn aiur_fn_N` per Aiur function. The generated code mirrors `src/aiur/execute.rs`'s QueryRecord side effects exactly: same `function_queries.insert` timing, same cache-hit multiplicity bumps, same memory-queries insertion order, same `bytes{1,2}_queries` updates, same IO sequencing. Per-witness trace hashes match the interpreter byte-for-byte on every const tested (`Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). * `Ix/Aiur/Stages/Codegen.lean`: structured Rust IR (`RustExpr` / `RustStmt` / `RustItem` / `MatchArm` / labeled blocks) plus a formatter. Codegen is a structural walk over the Bytecode AST, no string templating. `EmitM = StateM EmitState` threads `(nextVal, nextLabel)` so per-ValIdx Rust locals and per-MC labels are fresh. * Aiur's `map: Vec<G>` is gone in the generated kernel: every ValIdx becomes a Rust local `__v_{i}: G`. Match-arm bodies snapshot `nextVal` on entry so per-arm allocations don't leak to siblings. `MatchContinue` / `Yield` use labeled-block + `break 'label [G; OUT_SIZE]` to bubble yielded values out and rebind them at outer scope. * Deep Aiur recursion uses `stacker::maybe_grow(64 KiB, 4 MiB, …)` at the top of every generated fn, so the native call stack grows on demand rather than pre-reserving a giant thread stack. Verified against shard 24 of the 64-way `init.ixes` partition, which SEGFAULTs without it. * `ix codegen`: new CLI command. Compiles the IxVM Aiur source, walks the bytecode, writes the generated Rust to a fixed path (`src/ix/aiur_ixvm.rs`). The output path is hard-coded; no override. * `src/ix/aiur_ixvm.rs`: the generated kernel, 743 `aiur_fn_*` + `execute_generated` dispatch. Auto-regenerated; do not edit. `#![cfg_attr(rustfmt, rustfmt::skip)]` + a broad `#![allow(unused_*, non_snake_case, clippy::all)]` since the layout is for the compiler, not humans. * `src/ix/aiur_ixvm_runner.rs`: `execute_ixvm(toplevel, fun_idx, args, io_buffer) -> Result<(QueryRecord, Vec<G>), ExecError>`. Same return shape as `Toplevel::execute`, but routes through `execute_generated`. * `src/aiur/synthesis.rs`: `AiurSystem::prove_ixvm(...)` — same shape as `prove`, but the execute step calls `execute_ixvm`. Verification-compatible (proofs from one path verify under the other). * `src/aiur/execute.rs`: helpers exposed for the codegen'd kernel (`bytes{1,2}_execute` → `pub(crate)`, `unconstrained_big_uint_div_mod_helper` extracted, `CodegenBytes{1,2}{Op,}` aliases re-exported, `QueryRecord::new` → `pub(crate)`, `ExecError::InvalidFunIdx` added). * `src/ffi/aiur/protocol.rs`: two new FFI exports — `rs_aiur_toplevel_execute_ixvm` and `rs_aiur_system_prove_ixvm`. Same wire format as the existing `_execute` / `prove` exports. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.executeIxVM` — mirror of `execute`. * `Ix/Aiur/Protocol.lean`: `AiurSystem.proveIxVM`. * `Ix/Cli/CheckCmd.lean`: `runCompiled` now dispatches through `executeIxVM`. The Rust bytecode interpreter is no longer reachable from `ix check` (the Lean-side `--interp` fallback stays for richer error diagnostics). * `Ix/Cli/ProveCmd.lean`: `proveOne` uses `proveIxVM`. * `Cargo.toml`: `stacker = "0.1"`. For witnesses where execute is the dominant work (single-const checks via `ix check 'X'`), the codegen'd kernel is ~1.6× faster than the bytecode interpreter end-to-end (measured on `Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). Tiny consts are within ~10% (stacker probe overhead amortises). Peak RSS is within a few MB of the interpreter — stacker grows the stack on demand, no giant upfront reservation. For shard-driven `ix check --shard K` runs the speedup is invisible: per-shard wall time is dominated (~92%) by `buildShardCheckEnvWitness` in Lean, NOT by the kernel execute (~8%). Cutting witness construction is the next lever. `Nat.add_comm` proof produced via `proveIxVM` verifies under the existing `AiurSystem::verify`. End-to-end: lake exe ix prove 'Nat.add_comm' → proof addr 0d0ab0f9… lake exe ix verify 0d0ab0f9… → ok * Move shard witness construction to Rust + parallelise Replaces `IxVM.ClaimHarness.buildShardCheckEnvWitness` (Lean side, ~92% of shard wall time on heavy partitions) with a Rust port that builds the `aiur::execute::IOBuffer` directly, without per-byte boxing into Lean `Aiur.G` values. The two hot phases run on rayon: * Closure walk: each owned addr's transitive `Constant.refs` + projection-block traversal runs on its own thread; results are deduped through a `dashmap::DashSet`. * Byte-to-G conversion: per-const `(key, data)` tuples are built in parallel chunks of 256; final IOBuffer assembly (channel arena append + key→`IOKeyInfo` map insert) runs serially because the arena index is monotonic. * `src/ix/aiur_ixvm_witness.rs` (new): `build_shard_check_env_witness` produces `(claim, claim_digest_input, io_buffer)` ready to feed to `execute_ixvm`. Mirrors the 6-channel layout documented in `Ix/IxVM/ClaimHarness.lean` (claim / asm tree / const bytes / Defn hint / blob discriminator / blob raw bytes). * `src/ffi/aiur/protocol.rs`: new FFI `rs_aiur_toplevel_shard_check_ixvm` bundles witness build + `execute_ixvm` into one cross-language trip. Returns the same `(output, ioBuffer, queryCounts)` shape as `rs_aiur_toplevel_execute_ixvm`, so the Lean shim stays drop-in compatible. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.shardCheckIxVM` wraps the FFI. * `Ix/Cli/CheckCmd.lean`: shard-mode `ix check` (`--ixe + --ixes`) now dispatches through `runShardOwnedNative` → `shardCheckIxVM`, bypassing the Lean witness builder. Single- shard and whole-partition paths share the fast route; the legacy `runShardCheckManifest` / `runShardCheckAll` callbacks stay live for `--interp` only. Shard 26 of the 64-way `init.ixes` partition, on the same host: | Variant | total | speedup | | --- | --: | --: | | Lean witness + bytecode-interp (baseline)| 1029 s | 1.0× | | Lean witness + codegen kernel | 1015 s | 1.01× | | Rust witness (serial) + codegen | 101 s | 9.95× | | Rust witness (parallel) + codegen | 80 s | 12.9× | The serial Rust witness collapses the 935 s of Lean per-byte boxing into ~23 s; rayon hides that 23 s under the codegen-kernel execute (~78 s) so witness build effectively becomes free at the shard level. FFT cost matches the bytecode interpreter exactly (107_006_963_281), so the QueryRecord layout is bit-identical across all four variants. * Closure walk uses single-source BFS with the global visited set shared via `DashSet::contains` checks before pushing to the per-thread stack — avoids redundant work without a per-owned union step. * Per-channel ordering: each chunk's partial `ChannelEntries` feeds the serial fold in iteration order, so channel arena contents match the Lean side's exactly. Test still relies on trace-hash parity from the earlier codegen commit, which exercised the same path through the bytecode interpreter and the codegen kernel. * Coverage check is currently skipped on the IxVM-native whole-partition path (`runShardManifestAllNative`); legacy `runShardCheckAll` still does it for `--interp`. Re-enable separately if needed. * Codegen: value-returning byte ops, dead-fold the unconstrained plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed). * Wire Rust witness fast path through every check/prove entry point Per-claim mode (a `Claim.check addr none`) builds a closure rooted at the target address. That closure can be the whole environment when the target is a heavily-shared root constant. The Lean witness builder paid per-byte boxing into `Aiur.G` for every byte in that closure — the same cost we already eliminated for shard mode. # Surface Five new FFIs, all bundling witness build + execute_ixvm (and STARK prove where applicable) into one cross-language trip so the `IOBuffer` never crosses the boundary mid-pipeline: * `rs_aiur_toplevel_check_addr_ixvm` — `Bytecode.Toplevel.checkAddrIxVM (toplevel) (funIdx) (ixePath) (addrBytes)`: per-claim check; builds witness for `Claim.check addr none` via `build_claim_check_witness`, runs `execute_ixvm`. Same return shape as `shardCheckIxVM`. * `rs_aiur_toplevel_check_env_bytes_ixvm` — same as above but takes the env as a serialized byte blob (`Ixon.serEnv`) instead of a `.ixe` path. Used by the compiled-Lean-env code path (`ix check NAME` without `--ixe`), where the env is built in Lean memory. * `rs_aiur_system_prove_addr_ixvm` — `AiurSystem.proveAddrIxVM (...)`: per-claim prove (witness + execute + STARK prove all in Rust). * `rs_aiur_system_prove_env_bytes_ixvm` — bytes-blob counterpart. * `rs_aiur_system_shard_prove_ixvm` — `AiurSystem.shardProveIxVM (...)`: per-shard prove end-to-end in Rust. `build_claim_check_witness` lives next to `build_shard_check_env_witness` in `src/ix/aiur_ixvm_witness.rs` and uses the same parallel closure walk + parallel byte→G conversion. The bytes-blob FFIs additionally harvest `anon_hints` from each `Def` named entry after `Env::get` — `Env::get` (full form, used by the bytes-blob path) doesn't populate `env.anon_hints` the way `Env::get_anon` does, but the kernel's `verify_claim` reads ch 3 (Defn reducibility hints) so the hints must end up in the env. This mirrors the harvest pass already inside `get_anon` at `src/ix/ixon/serialize.rs:1683`. # Wiring `Ix.Cli.CheckCmd.WitnessSource`: a three-arm discriminated union threaded through `forEachClaim` (the shared check/prove driver): * `.native ixePath addr` — `.ixe`-backed env, Rust mmap. * `.nativeBytes envBytes addr` — Lean-memory env serialized to bytes, Rust decodes via `Env::get`. * `.lean witness` — pre-built `ClaimWitness` (fallback). `runCompiled` and `proveOne` dispatch on the source. # Coverage | Command | Path | |---------------------------------------|-----------------------------------------------| | `ix check NAME --ixe` | **`checkAddrIxVM`** (new) | | `ix check NAME` (no `--ixe`) | **`checkEnvBytesIxVM`** (new) | | `ix check --claim hex --ixe` | `checkAddrIxVM` for `check addr none` | | `ix check --ixes --shard K` | `shardCheckIxVM` (existed) | | `ix check --ixes` | `shardCheckIxVM` (existed) | | `ix prove NAME --ixe` | **`proveAddrIxVM`** (new) | | `ix prove NAME` (no `--ixe`) | **`proveEnvBytesIxVM`** (new) | | `ix prove --ixes --shard K` | **`shardProveIxVM`** (new) | | `ix prove --ixes` | **`shardProveIxVM`** loop (new) | | `Benchmarks/Typecheck.lean` Phase 1 | `checkAddrIxVM` / `executeIxVM` | | `Benchmarks/Typecheck.lean` Phase 2 | `proveAddrIxVM` / `proveIxVM` | | `Benchmarks/IxVM.lean` | `proveIxVM` (was `prove`) | `--interp` route is preserved: it materialises any `WitnessSource` back into a `ClaimWitness` before driving the Aiur source interpreter. `--claim <hex>` over non-`check addr none` variants (`eval` / `reveal` / `contains` / `checkEnv-with-asm`) still uses the Lean witness builder. # Sanity * `ix check Nat.add_comm --ixe init.ixe` (warm): 3.1 s wall, FFT = 23_603_449. * `ix check Nat.add_comm` (compiled env via bytes blob, warm): 2.4 s wall, FFT = 23_603_449. # Known overheads (tracked for a follow-up redesign) * **Per-claim env re-parse on `--ixe + many names`**: each FFI call re-runs `Env::get_anon_mmap` on the same path. Mathlib-scale iteration pays the lazy-index build N times instead of once. * **`--interp + .nativeBytes` round-trip**: Lean serializes the compiled env then `materialise` deserializes it back, just to feed `runInterp`. * **`runShardProveNative` claim reconstruct**: redoes the closure walk + canonical `AssumptionTree` build Lean-side after `shardProveIxVM` already computed the same claim internally. A future `EnvHandle`-based API would close all three: build the env once into a Rust-owned handle, pass the handle to every per-claim/per-shard FFI, and have prove FFIs return the serialized claim bytes. * EnvHandle: Rust-owned env shared across per-claim / per-shard FFI calls Before: every per-claim and per-shard FFI re-parsed the Ixon env (`Env::get_anon_mmap` per call). On `--ixe + many names` / all-shards prove, this is the dominant overhead for iteration-heavy workflows — O(num_consts) lazy-index build × N targets. After: the env lives once per CLI invocation in a Rust-owned `EnvHandle`. Lean holds an opaque `Aiur.EnvHandle` reference and threads `@& EnvHandle` through every per-target FFI call. The env is parsed exactly once at handle construction; downstream calls share it. # Surface Five new FFIs collapse the previous six per-call variants (`checkAddrIxVM`, `checkEnvBytesIxVM`, `shardCheckIxVM`, `proveAddrIxVM`, `proveEnvBytesIxVM`, `shardProveIxVM` — all deleted): * `rs_aiur_env_handle_from_ixe(path) → LeanExternal<EnvHandle>`: mmap-load via `Env::get_anon_mmap`. Anon parser already harvests `anon_hints`; no post-pass. * `rs_aiur_env_handle_from_bytes(blob) → LeanExternal<EnvHandle>`: decode `Ixon.serEnv`-shape blob via `Env::get` + harvest `anon_hints` from each `Def` named entry. Used by the compiled-Lean-env path. * `rs_aiur_toplevel_check_addr_with_env(toplevel, fun_idx, handle, addr_bytes)`: per-claim check. Reuses handle's parsed env. * `rs_aiur_toplevel_shard_check_with_env(toplevel, fun_idx, handle, owned_blob)`: per-shard check. * `rs_aiur_system_prove_addr_with_env(system, fri, fun_idx, handle, addr_bytes) → (claim_bytes, proof, ioBuffer)`: per-claim prove. Rust serializes the reconstructed `Ix.Claim` via `ixon::Claim::put` so Lean can deserialize via `Ixon.runGet Ix.Claim.get` directly — no closure walk + canonical `AssumptionTree` recomputation Lean-side. * `rs_aiur_system_shard_prove_with_env(system, fri, fun_idx, handle, owned_blob) → (claim_bytes, proof, ioBuffer)`: per-shard prove. # Lean * `Aiur.EnvHandle` opaque type with `fromIxe` / `fromBytes`. * `Aiur.Bytecode.Toplevel.checkAddrWithEnv` / `shardCheckWithEnv`, `Aiur.AiurSystem.proveAddrWithEnv` / `shardProveWithEnv`. * `Ix.Cli.CheckCmd.Target` replaces `WitnessSource`: ```lean inductive Target where | addr (a : Address) -- Claim.check addr none | shard (owned : Array Address) -- Claim.checkEnv | leanW (w : ClaimWitness) -- --interp, --claim non-check ``` `runCompiled` / `proveOne` take `(envHandle?, target)`. The envHandle is `none` only for `.leanW` (`--interp` legacy path). * `runShardOwnedNative` and the manifest-driver helpers take the envHandle as a parameter. Single-shard mode builds it once; all-shards mode reuses the same handle across every shard's FFI call (eliminates per-shard re-mmap). * `runShardProveNative` deserializes the wire claim bytes via `Ixon.runGet Ix.Claim.get` instead of re-running `shardCheckEnvClaim`. # Benchmarks `Benchmarks/Typecheck.lean` builds the envHandle once before Phase 1, reuses it across Phase 1 (execute) + Phase 2 (prove). Both phases now go through `checkAddrWithEnv` / `proveAddrWithEnv` on the full-closure path. `--subject-only` still uses Lean `buildVerifyConst` + `executeIxVM` / `proveIxVM` (witnesses intentionally small there). # New crate `crates/ix/src/env_handle.rs` — the `EnvHandle` struct + constructors live in the existing `ix` crate next to the witness builder and codegen runner. # Sanity * Multi-target check (warm): `ix check --ixe init.ixe Nat.add_comm Nat.add` → 3.3 s wall, single envHandle shared across both targets. * Shard 26 check (warm): `ix check --ixe init.ixe --ixes init.ixes --shard 26` → ~78 s wall, FFT = 107_006_963_281 (parity preserved). # Coverage | Command | Path | |--------------------------------------|-----------------------------------| | `ix check NAME --ixe` | `checkAddrWithEnv` | | `ix check NAME` (no `--ixe`) | `checkAddrWithEnv` + fromBytes | | `ix check --claim hex --ixe` | `checkAddrWithEnv` for check-none | | `ix check --ixes --shard K` | `shardCheckWithEnv` | | `ix check --ixes` | `shardCheckWithEnv` × all shards | | `ix prove NAME --ixe` | `proveAddrWithEnv` | | `ix prove --ixes --shard K` | `shardProveWithEnv` | | `ix prove --ixes` | `shardProveWithEnv` × all shards | | `Benchmarks/Typecheck` Phase 1+2 | `checkAddrWithEnv` / `proveAddrWithEnv` | Only `--interp` and `--claim hex` over a non-`check addr none` persisted claim still build a Lean `ClaimWitness`; both go via the `.leanW` target arm. * ix codegen: add --check flag for CI; gate stale generated kernel `ix codegen --check` compares the emitted Rust source against the on-disk `crates/ix/src/aiur_ixvm.rs` and exits 0 if identical, 1 otherwise. No write side effect. ~2 s warm; fast enough to gate on every PR. Wired into the `lean-test` CI job so a forgotten regen on a kernel-touching PR fails CI instead of merging stale generated code that drifts from the Bytecode → Rust emitter. * ix check: --interp {source|bytecode} to bypass codegen kernel Two interpreter modes behind a single `--interp` flag: * `--interp source`: Aiur source interpreter (`Aiur.runFunction` over the source-level `Decls`). Richer per-step error diagnostics. * `--interp bytecode`: generic Aiur bytecode interpreter (`Bytecode.Toplevel.execute`, `rs_aiur_toplevel_execute` route). Skips the `ix codegen` + `cargo build --release` cycle needed after editing `Ix/IxVM/*.lean` — the bytecode is rebuilt Lean-side at exe load. Slower per-check than the codegen kernel; ideal for tight iteration on the IxVM source. * omit the flag entirely for the native codegen kernel (default). Invalid values (e.g. `--interp foo`) fail fast with a clear error. # Plumbing The `check_addr_with_env` and `shard_check_with_env` FFIs take a `use_bytecode: bool` and dispatch via a shared `dispatch_execute` helper. `--interp bytecode` sets it to `true`. The `.leanW` arm of `runCompiled` also picks between `Bytecode.Toplevel.execute` and `executeIxVM` based on the same flag. The `--interp source` path requires a Lean `ClaimWitness`. The existing driver had switched to `.addr`/`.shard` targets against the Rust-owned `EnvHandle`; source-interp had no way to materialise those. `forEachClaim` now takes a `forceLeanWitness : Bool` — when true, it builds a Lean witness for every target (via `mkWitness` / `loadIxonEnv` for the compiled-Lean-env path) and passes `.leanW` so `runInterp` can consume it directly. # Sanity * `ix check Std.Time.Week.Offset.ofMilliseconds` (warm): 9.3 s wall, FFT = 12_430_516_949 (codegen kernel). * `ix check --interp bytecode Std.Time.Week.Offset.ofMilliseconds` (warm): 14.6 s wall, FFT = 12_430_516_949 (bytecode interp). * `ix check --interp source Eq` (warm): 6.3 s wall, output `Eq: ()`. * `ix check --interp garbage Nat.add_comm`: exits 1 with clear error. * IxVM codegen: regen + rebase-fixups after KLevel port Aligns the ap/codegen-ixvm-native stack with main's KLevel pointer port (b84a500) and re-lands clippy-clean under the workspace lint set: - Regen crates/ix/src/aiur_ixvm.rs from Aiur source (KLevel = &KLevelNode changes the generated fn shapes; ~26k lines net churn). - Extend the generated file's #![allow(...)] header (via Ix/Aiur/Stages/Codegen.lean) with clippy::ptr_as_ptr, clippy::match_same_arms, and clippy::large_types_passed_by_value — three lints the codegen's straight-line style trips deterministically on every regen and that clippy::all doesn't cover. - rustfmt reflow in crates/aiur/src/execute.rs and a handful of ix / ffi files that CI now enforces via the ap/aiur-aux-recursor-parity toolchain. - Fix 4 real clippy warnings uncovered under the workspace lint set: * env_handle.rs: clone-on-Copy → deref (ReducibilityHints is Copy). * aiur_ixvm_witness.rs: collapse if-let-if, drop needless continue, slice::from_ref instead of &[x.clone()]. * aiur_ixvm_runner.rs: keep execute_ixvm's Vec<G> arg (required by AiurSystem::prove_ixvm's fn-pointer bound) + local #[allow( clippy::needless_pass_by_value)] with a comment explaining why. * ffi/aiur/protocol.rs: unqualify aiur::G/IOBuffer/QueryRecord after the workspace lint added unused_qualifications; is_multiple_of() over % 0. Verified: cargo clippy --workspace --all-targets --all-features --D warnings clean; lake test -- --ignored ixvm passes. * ix-check + ixvm tests: coverage gate, --interp source fix, codegen parity + rename ix crate to ixvm-codegen Four review fixes on the codegen-ixvm-native stack. - **Coverage gate on \`--ixes\` (all-shards, native path).** \`runShardManifestAllNative\` now calls \`shardsCover\` before running any shard, matching the pre-refactor \`runShardCheckAll\` behavior. Without this, a stale/truncated \`.ixes\` manifest makes \`ix check --ixes\` exit 0 while some env constants are never checked by any shard — the mundane trigger being: edit a definition, rebuild the \`.ixe\`, forget to re-run \`ix shard\`, and the constants that go unchecked are exactly the ones just edited. Single-shard \`--shard K\` path is unchanged (consistent with pre-refactor). \`shardsCover\` moved above \`runShardManifestAllNative\` to satisfy forward-decl ordering. - **\`ix check --ixe … --claim <hex> --interp source\` regression.** The \`claimHex\` arm in \`forEachClaim\` ignored \`forceLeanWitness\` and always mapped \`.check addr none\` claims to \`Target.addr\`. Under \`--interp source\`, \`runInterp\` then rejects \`.addr\` with "\`--interp requires a Lean witness; addr/shard targets unreachable here\`". Now mirrors the sibling name-based arms and routes through the Lean witness path when \`forceLeanWitness\` is on. Only bit the common claim shape. - **Codegen ↔ bytecode parity CI test.** New \`runParityCase\` + \`parityCases\` in \`Tests/Ix/IxVM.lean\` wire every constant already listed in \`kernelCheckEntries\` plus \`kernel_unit_tests\` through BOTH \`Toplevel.execute\` (bytecode) and \`Toplevel.executeIxVM\` (codegen'd Rust), and diff the returned \`(output, IOBuffer, QueryCounts)\` triples. Turns "generated kernel ≡ interpreter on QueryRecord" from reviewed-by-hand into checked-by-CI. 129 assertions run per suite invocation (43 constants × output + IOBuffer + QueryCount), all pass on this branch head. - **Rename \`crates/ix\` → \`crates/ixvm-codegen\`.** The old \`ix\` crate held only the codegen'd IxVM kernel + its runner + witness helpers. \`ix-kernel\` is the hand-written Rust ix typechecker; \`ix-common\` / \`ix-compile\` are Ix-level infrastructure. \`ixvm-codegen\` cleanly names what this crate holds. Path updates: workspace \`members\` and \`workspace.dependencies\`, ffi crate dep + \`use\` sites, Lean \`codegenOutPath\` in \`Ix/Cli/CodegenCmd.lean\`, doc reference in \`Ix/Aiur/Semantics/BytecodeFfi.lean\`. \`ix codegen\` now writes to \`crates/ixvm-codegen/src/aiur_ixvm.rs\`. Verified: \`cargo check --workspace\` + \`lake test -- --ignored ixvm\` both exit 0 post-rename; parity + pinned FFT tests all pass. * cargo-deny: allow ar_archive_writer's Apache-2.0 WITH LLVM-exception Post-rename, ixvm-codegen pulls stacker → psm → ar_archive_writer (a build-time transitive). ar_archive_writer is licensed under the LLVM-exception variant of Apache-2.0 (standard for LLVM tooling; no additional runtime obligation for downstream users), which isn't on the workspace allow list. Add a per-crate exception scoped to ar_archive_writer so we don't blanket-allow the license family across the tree.

arthurpaulino · 2025-02-04T13:52:08Z

No description provided.

Lays the groundwork for measuring the kernel performance plan (plans/okay-let-s-write-a-lucky-dolphin.md) per audit §10. Counters live in a new kernel::perf module; KEnv carries a PerfCounters field that is dumped from a Drop impl when IX_PERF_COUNTERS=1 is set. Unset is the production default — every increment short-circuits via a LazyLock<bool> so the cost is a single cached branch on the hot path. Wired into the cache get sites the audit identified: - whnf_cache hits/misses (whnf.rs around 201/211) - whnf_no_delta_cache hits/misses (whnf.rs around 437/446) - infer_cache and infer_only_cache hits/misses (infer.rs around 45/51) - def_eq_cache hits/misses (def_eq.rs around 137/149) - def_eq_failure set hits/inserts (def_eq.rs around 360) - per-constant peak/avg MAX_REC_FUEL consumption (recorded in TypeChecker::reset before the next constant resets it) Verified against lake test -- kernel-check-env --ignored: 192156/192156 constants pass in 251s (4% over the 242s baseline; overhead is the LazyLock branch + atomic load in the disabled path). P1: def_rank_id replaces def_weight_id for hint-priority comparison Audit Tier 1 #3 (kernel-perf-adversarial-audit-2026-04-26.md, §4.2): the prior u32 encoding mapped Abbrev to u32::MAX-1 and saturating-added Regular(h) to h+1, which collide at h ≥ u32::MAX-2. When that happens the delta-direction logic treats Abbrev and a maximally-heavy Regular as "same height" and unfolds both, instead of preferring Abbrev as Lean does (compare(d_t->get_hints(), d_s->get_hints()) at type_checker.cpp:910). Replace the u32 weight with a (class: u8, height: u32) tuple compared lexicographically: - Opaque / Theorem / unknown → (0, 0) - Regular(h) → (1, h) (height ordering preserved within class) - Abbrev → (2, 0) (strictly above every Regular) Update the two call sites (is_def_eq lazy-delta height comparison and lazy_delta_step). The map_or default for "missing head" is preserved as (u8::MAX, u32::MAX) — the branch is dead in practice (a_delta && b_delta imply both heads are present) but kept consistent with the prior u32::MAX sentinel. Two regression tests: - def_rank_abbrev_above_saturated_regular: Abbrev outranks Regular(u32::MAX) (the previous saturation collision). - def_rank_regular_orders_by_height: height monotonically orders Regular ranks within the class. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 256s (no regression vs the 242s pre-perf-counters baseline; the +14s is from the IX_PERF_COUNTERS=unset LazyLock branch added in the prior commit, not this change). P2: peel_proj_forall fast-paths syntactic Pi in projection inference Audit Tier 1 #2 (kernel-perf-adversarial-audit-2026-04-26.md, §7.2): infer_proj's two parameter-consuming loops (param peel and field peel) called self.whnf(&r)? unconditionally per iteration, on a body mutated by subst at the previous step. The whnf cache rarely hits between iterations and each call re-traverses the substituted body. Extract a peel_proj_forall(&r, err) helper that: - tries ExprData::All(..) syntactically first (no WHNF call), and - falls back to full self.whnf(e) only when the binder isn't already syntactic Pi. This mirrors Lean's inferProj at type_checker.cpp:582–610. Both projection-inference loops now call peel_proj_forall instead of unconditional whnf. Behaviorally equivalent — same WHNF semantics on miss, no semantic change otherwise. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 258s (parity with the post-pre-work, post-P1 baseline; no measurable regression and the cache-hit-rate counters will move on tactic-heavy workloads under IX_PERF_COUNTERS=1). P3a: WhnfFlags substrate (no behavior change) Lays the foundation for the Lean4Lean architectural alignment described in plans/okay-let-s-write-a-lucky-dolphin.md. Phase 3a is substrate-only — no call site is migrated to cheap mode yet, so behavior is unchanged. Adds: - WhnfFlags { cheap_rec, cheap_proj } with FULL and CHEAP consts and is_full(). CHEAP is currently equal to FULL until Phase 3c wires it. - whnf_core_with_flags (private): the existing whnf_core impl, now threading flags into recursive calls and try_iota_with_flags. - whnf_core / whnf_core_cheap (super): FULL/CHEAP wrappers. - whnf_no_delta_with_flags (private): the existing whnf_no_delta impl with the Prj branch gated on cheap_proj — falls back to full whnf on the projected value when not cheap. - whnf_no_delta (pub) / whnf_no_delta_cheap (super): wrappers. - try_iota_with_flags: gates major-premise WHNF and string-literal constructor reduction on cheap_rec. - try_proj_app_reduce_with_flags: gates projected-value WHNF on cheap_proj. Cache reads/writes (whnf_no_delta_cache, equiv-manager second-chance) are gated on flags.is_full(): cheap callers neither read nor write the cache, preserving the invariant that any cached entry is a fully-reduced normal form. Phase 3b will inline the projection branch into whnf_core to match Lean4Lean's two-layer architecture (refs/lean4lean/Lean4Lean/ TypeChecker.lean:266, 297). Phase 3c will flip CHEAP to enable cheap_proj and migrate specific def-eq sites. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 243s (matches the 242s baseline; substrate adds no measurable overhead when CHEAP == FULL). P3b: inline projection into whnf_core (Lean4Lean architectural alignment) Move the Prj branch from whnf_no_delta_with_flags into whnf_core_with_flags so our whnf_core matches Lean4Lean's whnfCore semantics exactly (refs/lean4lean/Lean4Lean/TypeChecker.lean:284-292, 337-341). Before this commit: whnf_core — beta + zeta + iota + cheap projection (recursive whnf_core on val, no delta) whnf_no_delta — whnf_core + FULL projection (full whnf on val) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta Lean4Lean's architecture has no whnf_no_delta layer. Their whnfCore includes projection, with the cheap_proj flag deciding whether the projected value uses whnfCore (cheap) or whnf (full). After this commit: whnf_core (with WhnfFlags) — beta + zeta + iota + projection (cheap_proj controls val reduction) whnf_no_delta — whnf_core(_, FULL) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta The bare-Prj branch in whnf_no_delta_with_flags is removed — whnf_core now handles it directly. The App-of-Prj branch stays in whnf_no_delta because whnf_core's loop returns once the outermost Prj is resolved; try_proj_app_reduce_with_flags gives one more attempt at the same cheap_proj policy when the outer expression is App(Prj, ...). Pure refactor, no semantic change with CHEAP == FULL. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 270s (within noise of the 243s pre-refactor baseline). Phase 3c will flip CHEAP to enable cheap_proj=true and migrate def-eq's lazy-delta sites surgically per Lean4Lean's pattern. P3c (postponed): document the HeaderParsedSnapshot regression P3a (substrate) and P3b (Lean4Lean architectural alignment) are committed. Phase 3c — flipping CHEAP to enable cheap_proj=true and migrating the def-eq lazy-delta sites — was attempted but reproduced 5 failures on chained projections in Lean.Language.Lean.HeaderParsedSnapshot.* even after P3b inlined the projection branch into whnf_core. The substrate is left in place; CHEAP stays equal to FULL until the regression's root cause is understood. Notes on the regression for the next investigator: - Failures: HeaderParsedSnapshot.{stx,result?,metaSnap,toSnapshot,ictx}, all with 'projection: type mismatch with declared struct'. - The struct `extends` a parent, so each projection is a chained Prj whose val is itself a Prj into the parent. - The error comes from infer.rs's infer_proj at the head-vs-struct_id address compare, after FULL whnf on val_ty. That whnf is FULL, but val_ty was inferred via paths that may have consulted a def-eq cache populated under cheap mode. Possible cache-poisoning suspect: def_eq_cache writes a `false` result for inputs whose lazy-delta loop bottomed out under cheap projections. The cache key uses raw a/b hashes, not cheap-reduced shapes, so a stored `false` is indistinguishable from a FULL `false` by future readers. - Lean4Lean does not have an analogous wide def_eq_cache; their failure cache is keyed only on same-spine pairs in lazyDeltaReductionStep. Future P3c iterations should either prove the cache poisoning theory incorrect or restrict def_eq_cache writes to FULL-derived results.

* Refactor Claim ADT + add merkle infrastructure for ZK aggregation Adds the format hooks needed before recursive verification lands in Aiur. Five-variant Claim ADT with explicit assumption commitments, a canonical Blake3 merkle module, a serializable AssumptionTree for recovering leaf sets, and a Contains claim for the discharge step. Claim ADT (5 variants): - Eval { input, output, assumptions: Option<Address> } - Check { const_addr, assumptions: Option<Address> } - CheckEnv { root, assumptions: Option<Address> } - Reveal { comm, info } -- unchanged, no assumptions - Contains { tree, const_addr } -- new, for inclusion proofs Tag4 reorganized to keep everything in single-byte tags: - 0xE for env, comm, AssumptionTree, and claims (slots 0-7) - 0xF for proofs (slots 0-4; 5-7 reserved) - Comm moved from variant 5 -> 1 Matches the "Variant (0-7)" constraint documented in docs/Ixon.md. Env serialization: - Every .ixe file now carries a canonical merkle root over its consts.keys() in the on-disk header (non-optional, 32 bytes; empty const sets use the zero-address sentinel). - Two envs with the same const set produce byte-identical roots regardless of insertion order. Verified on deserialize. New modules: - src/ix/ixon/merkle.rs + Ix/Merkle.lean: canonical sorted builder, free-form merkle_join composition, membership proofs, domain separation per RFC 6962. - src/ix/ixon/assumption_tree.rs + Ix/AssumptionTree.lean: serializable merkle tree with Leaf/Padding/Node variants. canonical() builds the same shape merkle_root_canonical hashes; join() is O(1) free-form composition. - src/ix/kernel/claim.rs: builders that compute transitive-dep assumptions from an env (build_check_claim, build_eval_claim, build_check_env_claim, env_merkle_root). Other: - Extracted shared BFS walker on Env::bfs_refs + Env::transitive_deps_excl; the inlined test-feature copy in lean_env.rs now calls into it. - Lean FFI export rs_env_merkle_root for cross-impl verification of the env root. - Proof bytes for all variants are uniform opaque ZK bytes; witness data (e.g., Contains merkle paths) is prover-side scratch consumed by the ZK circuit and not transmitted on the wire. - docs/Ixon.md Tag4 tables and env section updated; .ixe extension documented. Recursive verification (the ZK proof generation for Contains and the aggregation discharge transitions) is intentionally deferred to a follow-up. Tests: 993 Rust unit tests pass (was 953 pre-refactor), 813 Lean tests pass with no failures; cargo clippy clean. * Add lazy env deserialization and anon-mode kernel FFI Two related changes to reduce kernel memory + lay groundwork for metadata-isolated typechecking. **Lazy constant deserialization.** `Env::consts` now stores `LazyConstant` (`Arc<[u8]>` + `OnceLock<Arc<Constant>>`) instead of `Constant`. Constants are materialized on first access and the structured form is cached. Sparse access patterns (single-constant typecheck, transitive_deps walk, claim builders) only parse the closure they need; non-reachable constants stay as raw bytes. The `.ixe` section-2 layout gains a Tag0 length sidecar before each constant's Tag4 bytes: old: [addr:32] [Tag4 constant bytes] new: [addr:32] [Tag0 length] [Tag4 constant bytes] The length is section-level framing, not part of the constant's content hash. `Address::hash(raw_bytes) == addr` is preserved. `Constant::put`/`Constant::get` are unchanged. The Lean side (`Ix.Ixon.putEnv`/`getEnv`) reads/writes the new format. **Anon-mode kernel FFI.** New `rs_kernel_check_consts_anon(path, addrs, quiet)` exposes anonymous-mode typechecking by content address. The kernel runs as `KEnv<Anon>` / `TypeChecker<Anon>` with every `M::MField<T>` erased to `()`, so the typechecking logic structurally cannot read metadata. Useful for zkPCC verifiers that hold only addresses. Supporting pieces: - `KernelMode::HAS_META: bool` for future compile-time gating. - `AnonEnv<'a>` wrapper (`src/ix/kernel/anon_env.rs`) exposing only consts/blobs/transitive walks — no `named`/`names`/`comms`. - `Env::get_anon` reads header + blobs + consts, parse-and-drops metadata sections (3-5), returns an `Env` with empty `named`/ `names`/`comms`. Same merkle-root verification as `Env::get`. - `rs_de_env_anon` FFI + `Ix.Ixon.rsDeEnvAnon` Lean wrapper. - `Ix.KernelCheck.rsCheckConstsAnonFFI` Lean binding. Caveat: `ixon_ingress::<Anon>` still consults `Env::named` internally to enumerate work items. The resulting `KEnv<Anon>` is metadata-free so the typechecker is anon, but full ingress-level metadata isolation is a follow-up. Tests: 16 new (9 lazy + sparsity; 3 AnonEnv; 4 get_anon). All 1009 Rust unit tests pass; all 813 Lean tests pass; clippy clean. * Simplify ix check + add metadata-free anon mode - `ix check` becomes .ixe-only with a positional `<path>`. Drops the `--lean` compile-and-check flow and the `--env` flag; direct Lean → kernel checks remain available through `rsCheckConstsFFI` for tests. Removes the redundant `CheckIxonCmd.lean`. - New `--anon` flag runs a metadata-free kernel check: loads the .ixe via `Env::get_anon` (discards named/names/comms), enumerates work items from `env.consts` alone, and rebuilds member + ctor projection addresses deterministically via `Constant::commit`. Rejects `--consts`/`--ns`/`--consts-file` since it always checks everything. - Compiler: unwrap singleton non-inductive Muts blocks to standalone Defn/Recr. `Expr::Rec(0, univs)` already resolves correctly against a one-member block, so the wrapper was pure overhead; this keeps the env structurally uniform and matches `compile_single_def`. The anon ingress for standalones passes `mut_ctx_override = [self_id]` so self-recursive standalones still typecheck. - Kernel: anon parallel runner mirrors `run_checks_parallel_on_large _stacks` (32 workers, slow-detection, RSS tracking). Genericized `check_one_const<M>` / `check_consts_loop<M>` / `format_tc_error<M>` to share the runner. `KernelMode::meta_field_with` / `meta_field_try` gate metadata lookups in expression ingress at compile time. - Anon lazy ingress (`LazyAnonIngress` in tc.rs, helpers in ingress.rs) dedupes blocks via `kenv.blocks.contains_key(&KId<Anon>(B, ()))` before re-running `ingress_anon_block` (prevents N² re-ingestion when sibling projections fault separately within one worker check). - `Env::get_anon` now harvests `ReducibilityHints` from the otherwise-discarded Named section into a sidecar `Env::anon_hints: FxHashMap<Address, ReducibilityHints>`. Anon ingress threads these through a new `hints_override` parameter on `ingress_defn` so the lazy-delta tiebreak in `def_eq::def_rank_id` sees realistic heights instead of `Regular(0)`. Hints are performance advice, not correctness data — supplying them in anon mode preserves the metadata-free trust model. - Regenerate the canonical addresses in `PrimAddrs::new()`; singleton- unwrap changed the content hash of every self-recursive primitive (Nat.add, String, Nat.rec, …). End-to-end on compileinitstd.ixe (105,487 constants): meta: 105487/105487 in 20.6s, peak RSS 7.4GB anon: 89010/89010 in 17.8s, peak RSS 4.0GB * Clean up cargo clippy --all-targets - 11× `cloned_ref_to_slice_refs`: `&[x.clone()]` → `std::slice::from_ref(&x)` in `assumption_tree.rs` + `merkle.rs` test modules. Same allocator footprint, no clone, matches the lint's recommended idiom. - 2× `useless_vec`: `vec![0xE3, 0x00, 0x00]` → `[0xE3, 0x00, 0x00]` in serde-reject tests where the buffer is only borrowed. All 1011 unit tests pass. `cargo clippy --all-targets` is now clean. * ix check: print full content hashes + add --workers flag - The anon-mode progress / failure-log labels and the meta-mode hash-display fallback were truncating addresses to 16 hex chars (`@1f4b195aefa10e26`). Drop the `[..16]` slice so the full 64-char Blake3 hex is printed (`@1f4b195aefa10e2690d13c5b98b3d9124d3fbb5c…`), matching what tooling like `--fail-out` records and what the metadata-free workflow actually needs to identify a constant. - `--workers N` flag on `ix check` plumbs through the existing `IX_KERNEL_CHECK_WORKERS` env var that `resolve_kernel_check_workers` in `src/ffi/kernel.rs` reads. Useful for isolating per-worker memory cost (`--workers 1` lets you see the env's static footprint without per-worker overhead) and for capping concurrency in resource-tight contexts. * Mmap .ixe + cache-free LazyConstant for anon mode `lake exe ix check compilemathlib.ixe --anon` on a 3.2 GB env now peaks at ~11 GB RSS, down from ~40 GB; build_anon_work is 50× faster. Three intertwined changes: 1. Memory-map the .ixe (avoids ~3.2 GB heap copy of bytes) - Cargo: add memmap2 = "0.9". - `BytesSource` enum in `src/ix/ixon/lazy.rs` distinguishes heap-resident `Arc<[u8]>` from `(Arc<Mmap>, offset, len)` windows into a memory-mapped file. `LazyConstant::raw_bytes`, `verify_address`, `PartialEq` route through `BytesSource::as_slice` so every existing consumer is mmap-transparent. - `Env::get_anon_mmap(path)` in `src/ix/ixon/serialize.rs` opens the file, mmaps it, and stores Section-2 consts as mmap-backed `LazyConstant`s. Sections 3-4 (names + named) are still parsed transiently to harvest hints, then dropped before return. - `rs_kernel_check_anon` (`src/ffi/kernel.rs`) switches to `get_anon_mmap`; the old `std::fs::read` + `Env::get_anon` heap path is gone for the anon FFI. 2. Strip the persistent `LazyConstant` parsed-Constant cache - `LazyConstant.cache` was `Arc<OnceLock<Arc<Constant>>>` and accumulated forever — for mathlib that meant ~30 GB of parsed `Arc<Expr>` trees pinned in the env across the entire run. - New shape: `cache: Option<Arc<Constant>>`. Populated only by `from_constant` (compile-side, where we already own the parsed value). `from_bytes`/`from_mmap_slice` set `None`, and their `get()` parses fresh on every call without storing. - Re-parse cost is bounded by the existing per-worker dedup: `kenv.consts` already addresses each ingressed constant once per work item, and `clear_releasing_memory()` drops the kenv between items. The kenv is now the only persistent materialization layer. 3. `LazyConstant::peek_variant` for `build_anon_work` - One-byte read of the outer Tag4 head to identify the `ConstantInfo` variant — no body parse, no allocation. - `build_anon_work` now dispatches on `peek_variant()`; only `Muts` blocks trigger `lc.get()` (we need the member list for projection-address enumeration), and that `Arc<Constant>` drops at the end of the match arm. - Previously every constant was fully materialized at startup just to read its variant tag. With ~95% of the env being standalone/projection, the work-enumeration pass now skims the env at near-IO speed. Test updates: - `mmap_slice_roundtrips` (already added with the earlier mmap scaffolding) exercises the mmap window roundtrip. - New `from_bytes_does_not_cache` and `from_constant_clones_share_cache` document the new caching contract. - `peek_variant_*` tests cover every variant + empty-bytes and unknown-flag error paths. - `lazy_sparsity_only_materializes_closure` in `src/ix/ixon/env.rs` reframed to assert BFS-of-closure correctness instead of cache side-effects (`is_materialized()` no longer fires for lazy loads). End-to-end on compilemathlib.ixe (--anon, 32 workers): before: 640658/640658 in 266s, peak RSS ~40 GB after: 640658/640658 in 223s, peak RSS 11.3 GB * Address review findings: correctness + small refactors Code-review pass over the anon-mode / mmap / cache-strip work. This commit lands the highest-value subset — the correctness fixes and the small refactors that prevent future drift — and leaves the larger items (format-version bump, sealed marker trait, AnonEnv audit) for separate follow-ups. Correctness: - Verify per-constant address on load (#1). `LazyConstant::verify_address` existed but was never invoked from `Env::get`, `get_anon`, or `get_anon_mmap`. The env-level merkle root catches missing/extra entries but not byte-tampering of a constant whose key is intact; without this check, corruption surfaced much later inside `LazyConstant::get` with a misleading parse error. Inline the `Address::hash(bytes) == addr` check in each loader's Section-2 loop. Added 3 corruption-detection tests (`env_const_bytes_tampering_*`). This required updating a handful of existing tests that stored constants under fake `Address::hash(b"a")` keys instead of their true content hashes — round-tripping such envs now correctly rejects. Added a `store_canonical(env, c) -> Address` test helper for the canonical pattern, and `*_discriminator(refs, n)` so tests can produce content-distinct constants when the same ref-set would otherwise collide. - Hard-error in `ixon_env_to_decoded` on parse failure (#6). `src/ffi/ixon/env.rs` used `filter_map` to silently drop any const whose bytes failed `LazyConstant::get` — the Lean caller had no signal of the lost entries. Switch to a Result-collecting loop and propagate the first parse error; update both FFI call sites (`rs_de_env`, `rs_de_env_anon`). - Doc fixes (#4, #5). `docs/Ixon.md`'s AssumptionTree section described only `Leaf=0x00` and `Node=0x01`, missing `Padding=0x01` (and the docs' `Node` tag collided with the real `Padding` tag — the real `Node` is `0x02`). Also: the env-section table cell said "opt-tagged merkle root" but the implementation writes a bare 32-byte address. Refactors that prevent future drift: - Canonical projection-address helpers (#9). Four production sites reconstructed `Constant::new(IxonCI::{D,I,R,C}Prj{...}).commit().0` by hand; the anon pipeline silently breaks if any one drifts. Extract `defn_proj_address` / `indc_proj_address` / `recr_proj_address` / `ctor_proj_address` (plus `_constant` variants) in `src/ix/ixon/constant.rs`. Update `compile.rs`, `compile/mutual.rs`, and the anon `*_proj_addr` helpers to call them. - `verify_proj_addr_in_env` helper (#21). `ingress_anon_block` had the same "computed-address not in env" check repeated four times (DPrj/RPrj/IPrj/CPrj). DRY into one helper that produces a consistent error format. UX: - Anon display labels switched to `#<hex>` everywhere (#18). Rust was emitting `@<hex>` in progress/fail-out; Lean's `runCheckAnon` was emitting `#{i}` (result index). Standardize on `#<hex>` so the CLI failure summary is joinable with the fail-out file. This required exposing addresses per result slot to Lean — `rsCheckAnonFFI` now returns `Array (String × Option CheckError)` pairing each result with its content-address hex string. Rust builds the pairs via a new `build_anon_result_array` helper. - Stale "check-ixon" header in FailureLog (#16). One-line rename in the doc comment + `# ix check-ixon failures` writeln to `# ix check failures`. Tests: - `testPrimitivesParity` (PrimAddrs regen-parity test). Catches silent drift between hardcoded primitive addresses in `PrimAddrs::new()` and what `rsCompileEnvFFI` produces from the live Lean primitives. Plumbing: new `PrimAddrs::lean_parity_table()` returns `(lean_name, hex)` pairs; new `rs_prim_addrs_canonical` FFI exposes them to Lean; new test in `BuildPrimitives.lean` iterates `kernelPrimitives`, compares against the hardcoded table, and fails with a printable diff on mismatch (with instructions to regenerate). Skipped: `eagerReduce` — synthetic kernel marker whose PrimAddrs value (`0xff…3`) intentionally diverges from its compiled content hash (which collides with `id`). - 3 new corruption-detection tests in `serialize.rs` exercise the Section-2 verify check for `Env::get`, `Env::get_anon`, `Env::get_anon_mmap`. Verification: `cargo test --lib` 1025 passing (3 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe` 105487/105487 in ~21s, `--anon` 89010/89010 with peak RSS 1.4 GB, `.lake/build/bin/IxTests --ignored rust-kernel-build-primitives` shows both `build primitives dump` and `primitive address parity (PrimAddrs vs live compile)` pass. Out of scope (deferred to follow-ups): - #2 Format version bump (Env::FLAG) - #3 rs_kernel_check_consts_anon still uses Env::get on main - #8 Sealed marker trait for the lazy_anon transmute - #11/#12 AnonEnv audit (vestigial wrapper) - #13, #15, #17, #19, #20, #22-25 Various quality cleanups * Round-2 review fixes: mmap defense + small cleanup Six small items from the round-2 review of the anon-mode work: - N1: stat-at-open defense in `Env::get_anon_mmap` (`src/ix/ixon/serialize.rs`). Capture the file size before mmap; if the kernel's mapped length disagrees (file truncated between open and map) bail with a clear error instead of letting workers SIGBUS deep in `LazyConstant::get`. Truncate-in-place under a live mapping is still undefined per POSIX — documented as a caller contract — but the open-time check catches the common case. - New `get_anon_mmap_survives_file_unlink` test: load via mmap, unlink the path, then materialize both already-touched and not-yet-touched constants. Locks in the inode-retention invariant that the SIGBUS analysis depends on; a future refactor that switched to `mmap_anonymous` or copied bytes into a tmpfile would fail loudly here instead of letting workers SIGBUS in production. - #11: delete `AnonEnv::as_ixon_env_unchecked` (`src/ix/kernel/anon_env.rs`). Dead `pub(crate)` escape hatch marked `#[allow(dead_code)]` — confirmed zero callers and removed. - #13: `debug_assert!` on `OnceLock::set` for both result vectors (`src/ffi/kernel.rs`, meta + anon paths). The previous `let _ = results[idx].set(...)` silently dropped a re-set. If a future `build_*_work` dedup refactor breaks the one-write-per-slot invariant, debug builds now panic with the slot index instead of silently losing results. - #19: `allNames.contains` O(n²) preflight → `Std.HashSet` lookup (`Ix/Cli/CheckCmd.lean`). At mathlib scale (~700k env names × thousands of seed names) the previous linear scan-per-name spent measurable seconds on missing-name preflight alone. - N4: doc imprecision in `src/ix/ixon/lazy.rs` — the cache-policy preamble said the worker `KEnv` is "cleared between work items" but the actual cadence is `clear_every` items (`IX_KERNEL_CHECK_CLEAR_EVERY`, default 1 but tunable). Refined wording so the doc matches the code. Verification: `cargo test --lib` 1026 passing (1 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe --anon` 89010/89010 in 15.2s peak RSS 1.4 GB (regression-clean). Out of scope, separate follow-ups: - #2 Format version bump - #3 rs_kernel_check_consts_anon zkPCC FFI (Env::get → mmap) - #8 Sealed marker trait for the lazy_anon transmute - #10 child_arena closure side-effect (widen MField to tuple) - #12 AnonEnv duplicates bfs_refs / transitive_deps_excl - #14 anon worker bypasses M-generic display helpers - #17 Anon --fail-out docstring claims --consts-file compat - #20 Singleton-unwrap duplicated across compile paths - #22-24 Lean `partial def`, `.get!` in tests, unguarded as-u64 casts - N2/N3 Lean is_materialized() callers + compute_const_size_breakdown - Test gaps: rs_kernel_check_anon integration, concurrent mmap, verify_address-on-mmap-corruption, etc. * rustfmt * Tests: derive RawConst addrs from content (verify_address fix) The `Address::hash(bytes) == addr` defense added in f792102 rejects `RawConst { addr, const }` pairs where `addr` is uncorrelated with `serConstant const`. The Rust-side tests in `serialize.rs` were updated at the time (`store_canonical` helper), but three Lean-side sites still produced mismatched pairs and surface now as: × ∃: Env serialization Lean==Rust × ∃: serde RawEnv with data × ∃: serde RawEnv roundtrip deserialization failed: rs_de_env: Env::get: const at idx 0 bytes hash to 6110b739… but stored under b177ec1b… Three fixes, all derive `addr := Address.blake3 (serConstant c)`: - `Tests/Gen/Ixon.lean::genRawConst` — property-test generator was `RawConst.mk <$> genAddress <*> genConstant` (uncorrelated). Now generates the const first, then derives addr from `serConstant`. - `Tests/FFI/Lifecycle.lean::serdeTests` (`withData` unit case) — was using one `testAddr := Address.blake3 #[1,2,3]` for both the const and unrelated blob/comm slots. Split into `testAddr` (still used for the content-hash-free blob/comm) and `testConstAddr := Address.blake3 (serConstant testConst)`. Added a name entry for the new canonical addr. - `Tests/FFI/Lifecycle.lean::genSerdeRawEnv` — the "pool of addresses" pattern picked const addrs from a random pool that couldn't possibly match content hashes. Restructured: consts derive canonical addrs from content, each gets its own name-table entry appended to the pool-derived entries so the serde pipeline's "all addresses resolvable" invariant still holds. * Regenerate Aiur primitive addresses for singleton-unwrap Compiler's singleton non-inductive Muts unwrap (12630aa) changed the content hashes of 48 primitives (`Nat.rec`, `Nat.add`, `Nat.mul`, …). The Aiur kernel's `Ix/IxVM/Kernel/Primitive.lean` still held the pre-unwrap blake3 bytes, so primitive dispatch missed every affected constant and fell through to structural `Nat.rec` unfolding. Tests like `IxVMPrim.nat_mul_big` then drove millions of Nat.succ unfoldings inside the Aiur trace, OOMing the prover. Addresses regenerated from `PrimAddrs::lean_parity_table()` (which the branch had already updated in `src/ix/kernel/primitive.rs`). * Fix standalone Recr ingress: typ-based inductive lookup `find_matching_block_addr` heuristic in the standalone-Recr branch of `build_convert_inputs_walk` picks the inductive block by matching ctor count to rule count. When multiple in-scope inductives share the same ctor count, it returns the wrong block — the stored rules' `ctor_idx` then points to a sibling's ctor, while `populate_rules` (canonical) derives `ctor_idx` from the right inductive's `ctor_indices`. The mismatch surfaces as `compare_rules`'s `assert_eq!(s_ctor, c_ctor)` panic (e.g. `38 != 40` for `IxVMPrim.nat_land_lit`). Switch the standalone-Recr path to the same typ-based resolution `build_aux_recr_ctor_idxs` already uses for aux Recr blocks: peel `params + motives + minors + indices` foralls of `recr.typ`, take the major's head Ref, look it up in `refs` to get the inductive's address, then resolve `IPrj.block` for the Muts wrapper. Slice ctor positions for the specific member via `extract_member_ctor_idxs`. Also store the resolved Muts `block_addr` in `CKRecr` (was passing the heuristic's wrong address through to `convert_recursor`). --------- Co-authored-by: Arthur Paulino <arthurleonardo.ap@gmail.com> Co-authored-by: Samuel Burnham <45365069+samuelburnham@users.noreply.github.com>

plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed).

* Aiur Bytecode → Rust codegen for the IxVM kernel Adds a new Aiur pipeline stage that translates `Bytecode.Toplevel` into a Rust source module, one `fn aiur_fn_N` per Aiur function. The generated code mirrors `src/aiur/execute.rs`'s QueryRecord side effects exactly: same `function_queries.insert` timing, same cache-hit multiplicity bumps, same memory-queries insertion order, same `bytes{1,2}_queries` updates, same IO sequencing. Per-witness trace hashes match the interpreter byte-for-byte on every const tested (`Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). * `Ix/Aiur/Stages/Codegen.lean`: structured Rust IR (`RustExpr` / `RustStmt` / `RustItem` / `MatchArm` / labeled blocks) plus a formatter. Codegen is a structural walk over the Bytecode AST, no string templating. `EmitM = StateM EmitState` threads `(nextVal, nextLabel)` so per-ValIdx Rust locals and per-MC labels are fresh. * Aiur's `map: Vec<G>` is gone in the generated kernel: every ValIdx becomes a Rust local `__v_{i}: G`. Match-arm bodies snapshot `nextVal` on entry so per-arm allocations don't leak to siblings. `MatchContinue` / `Yield` use labeled-block + `break 'label [G; OUT_SIZE]` to bubble yielded values out and rebind them at outer scope. * Deep Aiur recursion uses `stacker::maybe_grow(64 KiB, 4 MiB, …)` at the top of every generated fn, so the native call stack grows on demand rather than pre-reserving a giant thread stack. Verified against shard 24 of the 64-way `init.ixes` partition, which SEGFAULTs without it. * `ix codegen`: new CLI command. Compiles the IxVM Aiur source, walks the bytecode, writes the generated Rust to a fixed path (`src/ix/aiur_ixvm.rs`). The output path is hard-coded; no override. * `src/ix/aiur_ixvm.rs`: the generated kernel, 743 `aiur_fn_*` + `execute_generated` dispatch. Auto-regenerated; do not edit. `#![cfg_attr(rustfmt, rustfmt::skip)]` + a broad `#![allow(unused_*, non_snake_case, clippy::all)]` since the layout is for the compiler, not humans. * `src/ix/aiur_ixvm_runner.rs`: `execute_ixvm(toplevel, fun_idx, args, io_buffer) -> Result<(QueryRecord, Vec<G>), ExecError>`. Same return shape as `Toplevel::execute`, but routes through `execute_generated`. * `src/aiur/synthesis.rs`: `AiurSystem::prove_ixvm(...)` — same shape as `prove`, but the execute step calls `execute_ixvm`. Verification-compatible (proofs from one path verify under the other). * `src/aiur/execute.rs`: helpers exposed for the codegen'd kernel (`bytes{1,2}_execute` → `pub(crate)`, `unconstrained_big_uint_div_mod_helper` extracted, `CodegenBytes{1,2}{Op,}` aliases re-exported, `QueryRecord::new` → `pub(crate)`, `ExecError::InvalidFunIdx` added). * `src/ffi/aiur/protocol.rs`: two new FFI exports — `rs_aiur_toplevel_execute_ixvm` and `rs_aiur_system_prove_ixvm`. Same wire format as the existing `_execute` / `prove` exports. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.executeIxVM` — mirror of `execute`. * `Ix/Aiur/Protocol.lean`: `AiurSystem.proveIxVM`. * `Ix/Cli/CheckCmd.lean`: `runCompiled` now dispatches through `executeIxVM`. The Rust bytecode interpreter is no longer reachable from `ix check` (the Lean-side `--interp` fallback stays for richer error diagnostics). * `Ix/Cli/ProveCmd.lean`: `proveOne` uses `proveIxVM`. * `Cargo.toml`: `stacker = "0.1"`. For witnesses where execute is the dominant work (single-const checks via `ix check 'X'`), the codegen'd kernel is ~1.6× faster than the bytecode interpreter end-to-end (measured on `Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). Tiny consts are within ~10% (stacker probe overhead amortises). Peak RSS is within a few MB of the interpreter — stacker grows the stack on demand, no giant upfront reservation. For shard-driven `ix check --shard K` runs the speedup is invisible: per-shard wall time is dominated (~92%) by `buildShardCheckEnvWitness` in Lean, NOT by the kernel execute (~8%). Cutting witness construction is the next lever. `Nat.add_comm` proof produced via `proveIxVM` verifies under the existing `AiurSystem::verify`. End-to-end: lake exe ix prove 'Nat.add_comm' → proof addr 0d0ab0f9… lake exe ix verify 0d0ab0f9… → ok * Move shard witness construction to Rust + parallelise Replaces `IxVM.ClaimHarness.buildShardCheckEnvWitness` (Lean side, ~92% of shard wall time on heavy partitions) with a Rust port that builds the `aiur::execute::IOBuffer` directly, without per-byte boxing into Lean `Aiur.G` values. The two hot phases run on rayon: * Closure walk: each owned addr's transitive `Constant.refs` + projection-block traversal runs on its own thread; results are deduped through a `dashmap::DashSet`. * Byte-to-G conversion: per-const `(key, data)` tuples are built in parallel chunks of 256; final IOBuffer assembly (channel arena append + key→`IOKeyInfo` map insert) runs serially because the arena index is monotonic. * `src/ix/aiur_ixvm_witness.rs` (new): `build_shard_check_env_witness` produces `(claim, claim_digest_input, io_buffer)` ready to feed to `execute_ixvm`. Mirrors the 6-channel layout documented in `Ix/IxVM/ClaimHarness.lean` (claim / asm tree / const bytes / Defn hint / blob discriminator / blob raw bytes). * `src/ffi/aiur/protocol.rs`: new FFI `rs_aiur_toplevel_shard_check_ixvm` bundles witness build + `execute_ixvm` into one cross-language trip. Returns the same `(output, ioBuffer, queryCounts)` shape as `rs_aiur_toplevel_execute_ixvm`, so the Lean shim stays drop-in compatible. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.shardCheckIxVM` wraps the FFI. * `Ix/Cli/CheckCmd.lean`: shard-mode `ix check` (`--ixe + --ixes`) now dispatches through `runShardOwnedNative` → `shardCheckIxVM`, bypassing the Lean witness builder. Single- shard and whole-partition paths share the fast route; the legacy `runShardCheckManifest` / `runShardCheckAll` callbacks stay live for `--interp` only. Shard 26 of the 64-way `init.ixes` partition, on the same host: | Variant | total | speedup | | --- | --: | --: | | Lean witness + bytecode-interp (baseline)| 1029 s | 1.0× | | Lean witness + codegen kernel | 1015 s | 1.01× | | Rust witness (serial) + codegen | 101 s | 9.95× | | Rust witness (parallel) + codegen | 80 s | 12.9× | The serial Rust witness collapses the 935 s of Lean per-byte boxing into ~23 s; rayon hides that 23 s under the codegen-kernel execute (~78 s) so witness build effectively becomes free at the shard level. FFT cost matches the bytecode interpreter exactly (107_006_963_281), so the QueryRecord layout is bit-identical across all four variants. * Closure walk uses single-source BFS with the global visited set shared via `DashSet::contains` checks before pushing to the per-thread stack — avoids redundant work without a per-owned union step. * Per-channel ordering: each chunk's partial `ChannelEntries` feeds the serial fold in iteration order, so channel arena contents match the Lean side's exactly. Test still relies on trace-hash parity from the earlier codegen commit, which exercised the same path through the bytecode interpreter and the codegen kernel. * Coverage check is currently skipped on the IxVM-native whole-partition path (`runShardManifestAllNative`); legacy `runShardCheckAll` still does it for `--interp`. Re-enable separately if needed. * Codegen: value-returning byte ops, dead-fold the unconstrained plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed). * Wire Rust witness fast path through every check/prove entry point Per-claim mode (a `Claim.check addr none`) builds a closure rooted at the target address. That closure can be the whole environment when the target is a heavily-shared root constant. The Lean witness builder paid per-byte boxing into `Aiur.G` for every byte in that closure — the same cost we already eliminated for shard mode. # Surface Five new FFIs, all bundling witness build + execute_ixvm (and STARK prove where applicable) into one cross-language trip so the `IOBuffer` never crosses the boundary mid-pipeline: * `rs_aiur_toplevel_check_addr_ixvm` — `Bytecode.Toplevel.checkAddrIxVM (toplevel) (funIdx) (ixePath) (addrBytes)`: per-claim check; builds witness for `Claim.check addr none` via `build_claim_check_witness`, runs `execute_ixvm`. Same return shape as `shardCheckIxVM`. * `rs_aiur_toplevel_check_env_bytes_ixvm` — same as above but takes the env as a serialized byte blob (`Ixon.serEnv`) instead of a `.ixe` path. Used by the compiled-Lean-env code path (`ix check NAME` without `--ixe`), where the env is built in Lean memory. * `rs_aiur_system_prove_addr_ixvm` — `AiurSystem.proveAddrIxVM (...)`: per-claim prove (witness + execute + STARK prove all in Rust). * `rs_aiur_system_prove_env_bytes_ixvm` — bytes-blob counterpart. * `rs_aiur_system_shard_prove_ixvm` — `AiurSystem.shardProveIxVM (...)`: per-shard prove end-to-end in Rust. `build_claim_check_witness` lives next to `build_shard_check_env_witness` in `src/ix/aiur_ixvm_witness.rs` and uses the same parallel closure walk + parallel byte→G conversion. The bytes-blob FFIs additionally harvest `anon_hints` from each `Def` named entry after `Env::get` — `Env::get` (full form, used by the bytes-blob path) doesn't populate `env.anon_hints` the way `Env::get_anon` does, but the kernel's `verify_claim` reads ch 3 (Defn reducibility hints) so the hints must end up in the env. This mirrors the harvest pass already inside `get_anon` at `src/ix/ixon/serialize.rs:1683`. # Wiring `Ix.Cli.CheckCmd.WitnessSource`: a three-arm discriminated union threaded through `forEachClaim` (the shared check/prove driver): * `.native ixePath addr` — `.ixe`-backed env, Rust mmap. * `.nativeBytes envBytes addr` — Lean-memory env serialized to bytes, Rust decodes via `Env::get`. * `.lean witness` — pre-built `ClaimWitness` (fallback). `runCompiled` and `proveOne` dispatch on the source. # Coverage | Command | Path | |---------------------------------------|-----------------------------------------------| | `ix check NAME --ixe` | **`checkAddrIxVM`** (new) | | `ix check NAME` (no `--ixe`) | **`checkEnvBytesIxVM`** (new) | | `ix check --claim hex --ixe` | `checkAddrIxVM` for `check addr none` | | `ix check --ixes --shard K` | `shardCheckIxVM` (existed) | | `ix check --ixes` | `shardCheckIxVM` (existed) | | `ix prove NAME --ixe` | **`proveAddrIxVM`** (new) | | `ix prove NAME` (no `--ixe`) | **`proveEnvBytesIxVM`** (new) | | `ix prove --ixes --shard K` | **`shardProveIxVM`** (new) | | `ix prove --ixes` | **`shardProveIxVM`** loop (new) | | `Benchmarks/Typecheck.lean` Phase 1 | `checkAddrIxVM` / `executeIxVM` | | `Benchmarks/Typecheck.lean` Phase 2 | `proveAddrIxVM` / `proveIxVM` | | `Benchmarks/IxVM.lean` | `proveIxVM` (was `prove`) | `--interp` route is preserved: it materialises any `WitnessSource` back into a `ClaimWitness` before driving the Aiur source interpreter. `--claim <hex>` over non-`check addr none` variants (`eval` / `reveal` / `contains` / `checkEnv-with-asm`) still uses the Lean witness builder. # Sanity * `ix check Nat.add_comm --ixe init.ixe` (warm): 3.1 s wall, FFT = 23_603_449. * `ix check Nat.add_comm` (compiled env via bytes blob, warm): 2.4 s wall, FFT = 23_603_449. # Known overheads (tracked for a follow-up redesign) * **Per-claim env re-parse on `--ixe + many names`**: each FFI call re-runs `Env::get_anon_mmap` on the same path. Mathlib-scale iteration pays the lazy-index build N times instead of once. * **`--interp + .nativeBytes` round-trip**: Lean serializes the compiled env then `materialise` deserializes it back, just to feed `runInterp`. * **`runShardProveNative` claim reconstruct**: redoes the closure walk + canonical `AssumptionTree` build Lean-side after `shardProveIxVM` already computed the same claim internally. A future `EnvHandle`-based API would close all three: build the env once into a Rust-owned handle, pass the handle to every per-claim/per-shard FFI, and have prove FFIs return the serialized claim bytes. * EnvHandle: Rust-owned env shared across per-claim / per-shard FFI calls Before: every per-claim and per-shard FFI re-parsed the Ixon env (`Env::get_anon_mmap` per call). On `--ixe + many names` / all-shards prove, this is the dominant overhead for iteration-heavy workflows — O(num_consts) lazy-index build × N targets. After: the env lives once per CLI invocation in a Rust-owned `EnvHandle`. Lean holds an opaque `Aiur.EnvHandle` reference and threads `@& EnvHandle` through every per-target FFI call. The env is parsed exactly once at handle construction; downstream calls share it. # Surface Five new FFIs collapse the previous six per-call variants (`checkAddrIxVM`, `checkEnvBytesIxVM`, `shardCheckIxVM`, `proveAddrIxVM`, `proveEnvBytesIxVM`, `shardProveIxVM` — all deleted): * `rs_aiur_env_handle_from_ixe(path) → LeanExternal<EnvHandle>`: mmap-load via `Env::get_anon_mmap`. Anon parser already harvests `anon_hints`; no post-pass. * `rs_aiur_env_handle_from_bytes(blob) → LeanExternal<EnvHandle>`: decode `Ixon.serEnv`-shape blob via `Env::get` + harvest `anon_hints` from each `Def` named entry. Used by the compiled-Lean-env path. * `rs_aiur_toplevel_check_addr_with_env(toplevel, fun_idx, handle, addr_bytes)`: per-claim check. Reuses handle's parsed env. * `rs_aiur_toplevel_shard_check_with_env(toplevel, fun_idx, handle, owned_blob)`: per-shard check. * `rs_aiur_system_prove_addr_with_env(system, fri, fun_idx, handle, addr_bytes) → (claim_bytes, proof, ioBuffer)`: per-claim prove. Rust serializes the reconstructed `Ix.Claim` via `ixon::Claim::put` so Lean can deserialize via `Ixon.runGet Ix.Claim.get` directly — no closure walk + canonical `AssumptionTree` recomputation Lean-side. * `rs_aiur_system_shard_prove_with_env(system, fri, fun_idx, handle, owned_blob) → (claim_bytes, proof, ioBuffer)`: per-shard prove. # Lean * `Aiur.EnvHandle` opaque type with `fromIxe` / `fromBytes`. * `Aiur.Bytecode.Toplevel.checkAddrWithEnv` / `shardCheckWithEnv`, `Aiur.AiurSystem.proveAddrWithEnv` / `shardProveWithEnv`. * `Ix.Cli.CheckCmd.Target` replaces `WitnessSource`: ```lean inductive Target where | addr (a : Address) -- Claim.check addr none | shard (owned : Array Address) -- Claim.checkEnv | leanW (w : ClaimWitness) -- --interp, --claim non-check ``` `runCompiled` / `proveOne` take `(envHandle?, target)`. The envHandle is `none` only for `.leanW` (`--interp` legacy path). * `runShardOwnedNative` and the manifest-driver helpers take the envHandle as a parameter. Single-shard mode builds it once; all-shards mode reuses the same handle across every shard's FFI call (eliminates per-shard re-mmap). * `runShardProveNative` deserializes the wire claim bytes via `Ixon.runGet Ix.Claim.get` instead of re-running `shardCheckEnvClaim`. # Benchmarks `Benchmarks/Typecheck.lean` builds the envHandle once before Phase 1, reuses it across Phase 1 (execute) + Phase 2 (prove). Both phases now go through `checkAddrWithEnv` / `proveAddrWithEnv` on the full-closure path. `--subject-only` still uses Lean `buildVerifyConst` + `executeIxVM` / `proveIxVM` (witnesses intentionally small there). # New crate `crates/ix/src/env_handle.rs` — the `EnvHandle` struct + constructors live in the existing `ix` crate next to the witness builder and codegen runner. # Sanity * Multi-target check (warm): `ix check --ixe init.ixe Nat.add_comm Nat.add` → 3.3 s wall, single envHandle shared across both targets. * Shard 26 check (warm): `ix check --ixe init.ixe --ixes init.ixes --shard 26` → ~78 s wall, FFT = 107_006_963_281 (parity preserved). # Coverage | Command | Path | |--------------------------------------|-----------------------------------| | `ix check NAME --ixe` | `checkAddrWithEnv` | | `ix check NAME` (no `--ixe`) | `checkAddrWithEnv` + fromBytes | | `ix check --claim hex --ixe` | `checkAddrWithEnv` for check-none | | `ix check --ixes --shard K` | `shardCheckWithEnv` | | `ix check --ixes` | `shardCheckWithEnv` × all shards | | `ix prove NAME --ixe` | `proveAddrWithEnv` | | `ix prove --ixes --shard K` | `shardProveWithEnv` | | `ix prove --ixes` | `shardProveWithEnv` × all shards | | `Benchmarks/Typecheck` Phase 1+2 | `checkAddrWithEnv` / `proveAddrWithEnv` | Only `--interp` and `--claim hex` over a non-`check addr none` persisted claim still build a Lean `ClaimWitness`; both go via the `.leanW` target arm. * ix codegen: add --check flag for CI; gate stale generated kernel `ix codegen --check` compares the emitted Rust source against the on-disk `crates/ix/src/aiur_ixvm.rs` and exits 0 if identical, 1 otherwise. No write side effect. ~2 s warm; fast enough to gate on every PR. Wired into the `lean-test` CI job so a forgotten regen on a kernel-touching PR fails CI instead of merging stale generated code that drifts from the Bytecode → Rust emitter. * ix check: --interp {source|bytecode} to bypass codegen kernel Two interpreter modes behind a single `--interp` flag: * `--interp source`: Aiur source interpreter (`Aiur.runFunction` over the source-level `Decls`). Richer per-step error diagnostics. * `--interp bytecode`: generic Aiur bytecode interpreter (`Bytecode.Toplevel.execute`, `rs_aiur_toplevel_execute` route). Skips the `ix codegen` + `cargo build --release` cycle needed after editing `Ix/IxVM/*.lean` — the bytecode is rebuilt Lean-side at exe load. Slower per-check than the codegen kernel; ideal for tight iteration on the IxVM source. * omit the flag entirely for the native codegen kernel (default). Invalid values (e.g. `--interp foo`) fail fast with a clear error. # Plumbing The `check_addr_with_env` and `shard_check_with_env` FFIs take a `use_bytecode: bool` and dispatch via a shared `dispatch_execute` helper. `--interp bytecode` sets it to `true`. The `.leanW` arm of `runCompiled` also picks between `Bytecode.Toplevel.execute` and `executeIxVM` based on the same flag. The `--interp source` path requires a Lean `ClaimWitness`. The existing driver had switched to `.addr`/`.shard` targets against the Rust-owned `EnvHandle`; source-interp had no way to materialise those. `forEachClaim` now takes a `forceLeanWitness : Bool` — when true, it builds a Lean witness for every target (via `mkWitness` / `loadIxonEnv` for the compiled-Lean-env path) and passes `.leanW` so `runInterp` can consume it directly. # Sanity * `ix check Std.Time.Week.Offset.ofMilliseconds` (warm): 9.3 s wall, FFT = 12_430_516_949 (codegen kernel). * `ix check --interp bytecode Std.Time.Week.Offset.ofMilliseconds` (warm): 14.6 s wall, FFT = 12_430_516_949 (bytecode interp). * `ix check --interp source Eq` (warm): 6.3 s wall, output `Eq: ()`. * `ix check --interp garbage Nat.add_comm`: exits 1 with clear error. * IxVM codegen: regen + rebase-fixups after KLevel port Aligns the ap/codegen-ixvm-native stack with main's KLevel pointer port (b84a500) and re-lands clippy-clean under the workspace lint set: - Regen crates/ix/src/aiur_ixvm.rs from Aiur source (KLevel = &KLevelNode changes the generated fn shapes; ~26k lines net churn). - Extend the generated file's #![allow(...)] header (via Ix/Aiur/Stages/Codegen.lean) with clippy::ptr_as_ptr, clippy::match_same_arms, and clippy::large_types_passed_by_value — three lints the codegen's straight-line style trips deterministically on every regen and that clippy::all doesn't cover. - rustfmt reflow in crates/aiur/src/execute.rs and a handful of ix / ffi files that CI now enforces via the ap/aiur-aux-recursor-parity toolchain. - Fix 4 real clippy warnings uncovered under the workspace lint set: * env_handle.rs: clone-on-Copy → deref (ReducibilityHints is Copy). * aiur_ixvm_witness.rs: collapse if-let-if, drop needless continue, slice::from_ref instead of &[x.clone()]. * aiur_ixvm_runner.rs: keep execute_ixvm's Vec<G> arg (required by AiurSystem::prove_ixvm's fn-pointer bound) + local #[allow( clippy::needless_pass_by_value)] with a comment explaining why. * ffi/aiur/protocol.rs: unqualify aiur::G/IOBuffer/QueryRecord after the workspace lint added unused_qualifications; is_multiple_of() over % 0. Verified: cargo clippy --workspace --all-targets --all-features --D warnings clean; lake test -- --ignored ixvm passes. * ix-check + ixvm tests: coverage gate, --interp source fix, codegen parity + rename ix crate to ixvm-codegen Four review fixes on the codegen-ixvm-native stack. - **Coverage gate on \`--ixes\` (all-shards, native path).** \`runShardManifestAllNative\` now calls \`shardsCover\` before running any shard, matching the pre-refactor \`runShardCheckAll\` behavior. Without this, a stale/truncated \`.ixes\` manifest makes \`ix check --ixes\` exit 0 while some env constants are never checked by any shard — the mundane trigger being: edit a definition, rebuild the \`.ixe\`, forget to re-run \`ix shard\`, and the constants that go unchecked are exactly the ones just edited. Single-shard \`--shard K\` path is unchanged (consistent with pre-refactor). \`shardsCover\` moved above \`runShardManifestAllNative\` to satisfy forward-decl ordering. - **\`ix check --ixe … --claim <hex> --interp source\` regression.** The \`claimHex\` arm in \`forEachClaim\` ignored \`forceLeanWitness\` and always mapped \`.check addr none\` claims to \`Target.addr\`. Under \`--interp source\`, \`runInterp\` then rejects \`.addr\` with "\`--interp requires a Lean witness; addr/shard targets unreachable here\`". Now mirrors the sibling name-based arms and routes through the Lean witness path when \`forceLeanWitness\` is on. Only bit the common claim shape. - **Codegen ↔ bytecode parity CI test.** New \`runParityCase\` + \`parityCases\` in \`Tests/Ix/IxVM.lean\` wire every constant already listed in \`kernelCheckEntries\` plus \`kernel_unit_tests\` through BOTH \`Toplevel.execute\` (bytecode) and \`Toplevel.executeIxVM\` (codegen'd Rust), and diff the returned \`(output, IOBuffer, QueryCounts)\` triples. Turns "generated kernel ≡ interpreter on QueryRecord" from reviewed-by-hand into checked-by-CI. 129 assertions run per suite invocation (43 constants × output + IOBuffer + QueryCount), all pass on this branch head. - **Rename \`crates/ix\` → \`crates/ixvm-codegen\`.** The old \`ix\` crate held only the codegen'd IxVM kernel + its runner + witness helpers. \`ix-kernel\` is the hand-written Rust ix typechecker; \`ix-common\` / \`ix-compile\` are Ix-level infrastructure. \`ixvm-codegen\` cleanly names what this crate holds. Path updates: workspace \`members\` and \`workspace.dependencies\`, ffi crate dep + \`use\` sites, Lean \`codegenOutPath\` in \`Ix/Cli/CodegenCmd.lean\`, doc reference in \`Ix/Aiur/Semantics/BytecodeFfi.lean\`. \`ix codegen\` now writes to \`crates/ixvm-codegen/src/aiur_ixvm.rs\`. Verified: \`cargo check --workspace\` + \`lake test -- --ignored ixvm\` both exit 0 post-rename; parity + pinned FFT tests all pass. * cargo-deny: allow ar_archive_writer's Apache-2.0 WITH LLVM-exception Post-rename, ixvm-codegen pulls stacker → psm → ar_archive_writer (a build-time transitive). ar_archive_writer is licensed under the LLVM-exception variant of Apache-2.0 (standard for LLVM tooling; no additional runtime obligation for downstream users), which isn't on the workspace allow list. Add a per-crate exception scoped to ar_archive_writer so we don't blanket-allow the license family across the tree.

arthurpaulino · 2025-02-04T13:52:08Z

No description provided.

Lays the groundwork for measuring the kernel performance plan (plans/okay-let-s-write-a-lucky-dolphin.md) per audit §10. Counters live in a new kernel::perf module; KEnv carries a PerfCounters field that is dumped from a Drop impl when IX_PERF_COUNTERS=1 is set. Unset is the production default — every increment short-circuits via a LazyLock<bool> so the cost is a single cached branch on the hot path. Wired into the cache get sites the audit identified: - whnf_cache hits/misses (whnf.rs around 201/211) - whnf_no_delta_cache hits/misses (whnf.rs around 437/446) - infer_cache and infer_only_cache hits/misses (infer.rs around 45/51) - def_eq_cache hits/misses (def_eq.rs around 137/149) - def_eq_failure set hits/inserts (def_eq.rs around 360) - per-constant peak/avg MAX_REC_FUEL consumption (recorded in TypeChecker::reset before the next constant resets it) Verified against lake test -- kernel-check-env --ignored: 192156/192156 constants pass in 251s (4% over the 242s baseline; overhead is the LazyLock branch + atomic load in the disabled path). P1: def_rank_id replaces def_weight_id for hint-priority comparison Audit Tier 1 #3 (kernel-perf-adversarial-audit-2026-04-26.md, §4.2): the prior u32 encoding mapped Abbrev to u32::MAX-1 and saturating-added Regular(h) to h+1, which collide at h ≥ u32::MAX-2. When that happens the delta-direction logic treats Abbrev and a maximally-heavy Regular as "same height" and unfolds both, instead of preferring Abbrev as Lean does (compare(d_t->get_hints(), d_s->get_hints()) at type_checker.cpp:910). Replace the u32 weight with a (class: u8, height: u32) tuple compared lexicographically: - Opaque / Theorem / unknown → (0, 0) - Regular(h) → (1, h) (height ordering preserved within class) - Abbrev → (2, 0) (strictly above every Regular) Update the two call sites (is_def_eq lazy-delta height comparison and lazy_delta_step). The map_or default for "missing head" is preserved as (u8::MAX, u32::MAX) — the branch is dead in practice (a_delta && b_delta imply both heads are present) but kept consistent with the prior u32::MAX sentinel. Two regression tests: - def_rank_abbrev_above_saturated_regular: Abbrev outranks Regular(u32::MAX) (the previous saturation collision). - def_rank_regular_orders_by_height: height monotonically orders Regular ranks within the class. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 256s (no regression vs the 242s pre-perf-counters baseline; the +14s is from the IX_PERF_COUNTERS=unset LazyLock branch added in the prior commit, not this change). P2: peel_proj_forall fast-paths syntactic Pi in projection inference Audit Tier 1 #2 (kernel-perf-adversarial-audit-2026-04-26.md, §7.2): infer_proj's two parameter-consuming loops (param peel and field peel) called self.whnf(&r)? unconditionally per iteration, on a body mutated by subst at the previous step. The whnf cache rarely hits between iterations and each call re-traverses the substituted body. Extract a peel_proj_forall(&r, err) helper that: - tries ExprData::All(..) syntactically first (no WHNF call), and - falls back to full self.whnf(e) only when the binder isn't already syntactic Pi. This mirrors Lean's inferProj at type_checker.cpp:582–610. Both projection-inference loops now call peel_proj_forall instead of unconditional whnf. Behaviorally equivalent — same WHNF semantics on miss, no semantic change otherwise. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 258s (parity with the post-pre-work, post-P1 baseline; no measurable regression and the cache-hit-rate counters will move on tactic-heavy workloads under IX_PERF_COUNTERS=1). P3a: WhnfFlags substrate (no behavior change) Lays the foundation for the Lean4Lean architectural alignment described in plans/okay-let-s-write-a-lucky-dolphin.md. Phase 3a is substrate-only — no call site is migrated to cheap mode yet, so behavior is unchanged. Adds: - WhnfFlags { cheap_rec, cheap_proj } with FULL and CHEAP consts and is_full(). CHEAP is currently equal to FULL until Phase 3c wires it. - whnf_core_with_flags (private): the existing whnf_core impl, now threading flags into recursive calls and try_iota_with_flags. - whnf_core / whnf_core_cheap (super): FULL/CHEAP wrappers. - whnf_no_delta_with_flags (private): the existing whnf_no_delta impl with the Prj branch gated on cheap_proj — falls back to full whnf on the projected value when not cheap. - whnf_no_delta (pub) / whnf_no_delta_cheap (super): wrappers. - try_iota_with_flags: gates major-premise WHNF and string-literal constructor reduction on cheap_rec. - try_proj_app_reduce_with_flags: gates projected-value WHNF on cheap_proj. Cache reads/writes (whnf_no_delta_cache, equiv-manager second-chance) are gated on flags.is_full(): cheap callers neither read nor write the cache, preserving the invariant that any cached entry is a fully-reduced normal form. Phase 3b will inline the projection branch into whnf_core to match Lean4Lean's two-layer architecture (refs/lean4lean/Lean4Lean/ TypeChecker.lean:266, 297). Phase 3c will flip CHEAP to enable cheap_proj and migrate specific def-eq sites. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 243s (matches the 242s baseline; substrate adds no measurable overhead when CHEAP == FULL). P3b: inline projection into whnf_core (Lean4Lean architectural alignment) Move the Prj branch from whnf_no_delta_with_flags into whnf_core_with_flags so our whnf_core matches Lean4Lean's whnfCore semantics exactly (refs/lean4lean/Lean4Lean/TypeChecker.lean:284-292, 337-341). Before this commit: whnf_core — beta + zeta + iota + cheap projection (recursive whnf_core on val, no delta) whnf_no_delta — whnf_core + FULL projection (full whnf on val) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta Lean4Lean's architecture has no whnf_no_delta layer. Their whnfCore includes projection, with the cheap_proj flag deciding whether the projected value uses whnfCore (cheap) or whnf (full). After this commit: whnf_core (with WhnfFlags) — beta + zeta + iota + projection (cheap_proj controls val reduction) whnf_no_delta — whnf_core(_, FULL) + native primitives + projection_definition + quotient whnf — whnf_no_delta + delta The bare-Prj branch in whnf_no_delta_with_flags is removed — whnf_core now handles it directly. The App-of-Prj branch stays in whnf_no_delta because whnf_core's loop returns once the outermost Prj is resolved; try_proj_app_reduce_with_flags gives one more attempt at the same cheap_proj policy when the outer expression is App(Prj, ...). Pure refactor, no semantic change with CHEAP == FULL. Verified with lake test -- kernel-check-env --ignored: 192156/192156 in 270s (within noise of the 243s pre-refactor baseline). Phase 3c will flip CHEAP to enable cheap_proj=true and migrate def-eq's lazy-delta sites surgically per Lean4Lean's pattern. P3c (postponed): document the HeaderParsedSnapshot regression P3a (substrate) and P3b (Lean4Lean architectural alignment) are committed. Phase 3c — flipping CHEAP to enable cheap_proj=true and migrating the def-eq lazy-delta sites — was attempted but reproduced 5 failures on chained projections in Lean.Language.Lean.HeaderParsedSnapshot.* even after P3b inlined the projection branch into whnf_core. The substrate is left in place; CHEAP stays equal to FULL until the regression's root cause is understood. Notes on the regression for the next investigator: - Failures: HeaderParsedSnapshot.{stx,result?,metaSnap,toSnapshot,ictx}, all with 'projection: type mismatch with declared struct'. - The struct `extends` a parent, so each projection is a chained Prj whose val is itself a Prj into the parent. - The error comes from infer.rs's infer_proj at the head-vs-struct_id address compare, after FULL whnf on val_ty. That whnf is FULL, but val_ty was inferred via paths that may have consulted a def-eq cache populated under cheap mode. Possible cache-poisoning suspect: def_eq_cache writes a `false` result for inputs whose lazy-delta loop bottomed out under cheap projections. The cache key uses raw a/b hashes, not cheap-reduced shapes, so a stored `false` is indistinguishable from a FULL `false` by future readers. - Lean4Lean does not have an analogous wide def_eq_cache; their failure cache is keyed only on same-spine pairs in lazyDeltaReductionStep. Future P3c iterations should either prove the cache poisoning theory incorrect or restrict def_eq_cache writes to FULL-derived results.

* Refactor Claim ADT + add merkle infrastructure for ZK aggregation Adds the format hooks needed before recursive verification lands in Aiur. Five-variant Claim ADT with explicit assumption commitments, a canonical Blake3 merkle module, a serializable AssumptionTree for recovering leaf sets, and a Contains claim for the discharge step. Claim ADT (5 variants): - Eval { input, output, assumptions: Option<Address> } - Check { const_addr, assumptions: Option<Address> } - CheckEnv { root, assumptions: Option<Address> } - Reveal { comm, info } -- unchanged, no assumptions - Contains { tree, const_addr } -- new, for inclusion proofs Tag4 reorganized to keep everything in single-byte tags: - 0xE for env, comm, AssumptionTree, and claims (slots 0-7) - 0xF for proofs (slots 0-4; 5-7 reserved) - Comm moved from variant 5 -> 1 Matches the "Variant (0-7)" constraint documented in docs/Ixon.md. Env serialization: - Every .ixe file now carries a canonical merkle root over its consts.keys() in the on-disk header (non-optional, 32 bytes; empty const sets use the zero-address sentinel). - Two envs with the same const set produce byte-identical roots regardless of insertion order. Verified on deserialize. New modules: - src/ix/ixon/merkle.rs + Ix/Merkle.lean: canonical sorted builder, free-form merkle_join composition, membership proofs, domain separation per RFC 6962. - src/ix/ixon/assumption_tree.rs + Ix/AssumptionTree.lean: serializable merkle tree with Leaf/Padding/Node variants. canonical() builds the same shape merkle_root_canonical hashes; join() is O(1) free-form composition. - src/ix/kernel/claim.rs: builders that compute transitive-dep assumptions from an env (build_check_claim, build_eval_claim, build_check_env_claim, env_merkle_root). Other: - Extracted shared BFS walker on Env::bfs_refs + Env::transitive_deps_excl; the inlined test-feature copy in lean_env.rs now calls into it. - Lean FFI export rs_env_merkle_root for cross-impl verification of the env root. - Proof bytes for all variants are uniform opaque ZK bytes; witness data (e.g., Contains merkle paths) is prover-side scratch consumed by the ZK circuit and not transmitted on the wire. - docs/Ixon.md Tag4 tables and env section updated; .ixe extension documented. Recursive verification (the ZK proof generation for Contains and the aggregation discharge transitions) is intentionally deferred to a follow-up. Tests: 993 Rust unit tests pass (was 953 pre-refactor), 813 Lean tests pass with no failures; cargo clippy clean. * Add lazy env deserialization and anon-mode kernel FFI Two related changes to reduce kernel memory + lay groundwork for metadata-isolated typechecking. **Lazy constant deserialization.** `Env::consts` now stores `LazyConstant` (`Arc<[u8]>` + `OnceLock<Arc<Constant>>`) instead of `Constant`. Constants are materialized on first access and the structured form is cached. Sparse access patterns (single-constant typecheck, transitive_deps walk, claim builders) only parse the closure they need; non-reachable constants stay as raw bytes. The `.ixe` section-2 layout gains a Tag0 length sidecar before each constant's Tag4 bytes: old: [addr:32] [Tag4 constant bytes] new: [addr:32] [Tag0 length] [Tag4 constant bytes] The length is section-level framing, not part of the constant's content hash. `Address::hash(raw_bytes) == addr` is preserved. `Constant::put`/`Constant::get` are unchanged. The Lean side (`Ix.Ixon.putEnv`/`getEnv`) reads/writes the new format. **Anon-mode kernel FFI.** New `rs_kernel_check_consts_anon(path, addrs, quiet)` exposes anonymous-mode typechecking by content address. The kernel runs as `KEnv<Anon>` / `TypeChecker<Anon>` with every `M::MField<T>` erased to `()`, so the typechecking logic structurally cannot read metadata. Useful for zkPCC verifiers that hold only addresses. Supporting pieces: - `KernelMode::HAS_META: bool` for future compile-time gating. - `AnonEnv<'a>` wrapper (`src/ix/kernel/anon_env.rs`) exposing only consts/blobs/transitive walks — no `named`/`names`/`comms`. - `Env::get_anon` reads header + blobs + consts, parse-and-drops metadata sections (3-5), returns an `Env` with empty `named`/ `names`/`comms`. Same merkle-root verification as `Env::get`. - `rs_de_env_anon` FFI + `Ix.Ixon.rsDeEnvAnon` Lean wrapper. - `Ix.KernelCheck.rsCheckConstsAnonFFI` Lean binding. Caveat: `ixon_ingress::<Anon>` still consults `Env::named` internally to enumerate work items. The resulting `KEnv<Anon>` is metadata-free so the typechecker is anon, but full ingress-level metadata isolation is a follow-up. Tests: 16 new (9 lazy + sparsity; 3 AnonEnv; 4 get_anon). All 1009 Rust unit tests pass; all 813 Lean tests pass; clippy clean. * Simplify ix check + add metadata-free anon mode - `ix check` becomes .ixe-only with a positional `<path>`. Drops the `--lean` compile-and-check flow and the `--env` flag; direct Lean → kernel checks remain available through `rsCheckConstsFFI` for tests. Removes the redundant `CheckIxonCmd.lean`. - New `--anon` flag runs a metadata-free kernel check: loads the .ixe via `Env::get_anon` (discards named/names/comms), enumerates work items from `env.consts` alone, and rebuilds member + ctor projection addresses deterministically via `Constant::commit`. Rejects `--consts`/`--ns`/`--consts-file` since it always checks everything. - Compiler: unwrap singleton non-inductive Muts blocks to standalone Defn/Recr. `Expr::Rec(0, univs)` already resolves correctly against a one-member block, so the wrapper was pure overhead; this keeps the env structurally uniform and matches `compile_single_def`. The anon ingress for standalones passes `mut_ctx_override = [self_id]` so self-recursive standalones still typecheck. - Kernel: anon parallel runner mirrors `run_checks_parallel_on_large _stacks` (32 workers, slow-detection, RSS tracking). Genericized `check_one_const<M>` / `check_consts_loop<M>` / `format_tc_error<M>` to share the runner. `KernelMode::meta_field_with` / `meta_field_try` gate metadata lookups in expression ingress at compile time. - Anon lazy ingress (`LazyAnonIngress` in tc.rs, helpers in ingress.rs) dedupes blocks via `kenv.blocks.contains_key(&KId<Anon>(B, ()))` before re-running `ingress_anon_block` (prevents N² re-ingestion when sibling projections fault separately within one worker check). - `Env::get_anon` now harvests `ReducibilityHints` from the otherwise-discarded Named section into a sidecar `Env::anon_hints: FxHashMap<Address, ReducibilityHints>`. Anon ingress threads these through a new `hints_override` parameter on `ingress_defn` so the lazy-delta tiebreak in `def_eq::def_rank_id` sees realistic heights instead of `Regular(0)`. Hints are performance advice, not correctness data — supplying them in anon mode preserves the metadata-free trust model. - Regenerate the canonical addresses in `PrimAddrs::new()`; singleton- unwrap changed the content hash of every self-recursive primitive (Nat.add, String, Nat.rec, …). End-to-end on compileinitstd.ixe (105,487 constants): meta: 105487/105487 in 20.6s, peak RSS 7.4GB anon: 89010/89010 in 17.8s, peak RSS 4.0GB * Clean up cargo clippy --all-targets - 11× `cloned_ref_to_slice_refs`: `&[x.clone()]` → `std::slice::from_ref(&x)` in `assumption_tree.rs` + `merkle.rs` test modules. Same allocator footprint, no clone, matches the lint's recommended idiom. - 2× `useless_vec`: `vec![0xE3, 0x00, 0x00]` → `[0xE3, 0x00, 0x00]` in serde-reject tests where the buffer is only borrowed. All 1011 unit tests pass. `cargo clippy --all-targets` is now clean. * ix check: print full content hashes + add --workers flag - The anon-mode progress / failure-log labels and the meta-mode hash-display fallback were truncating addresses to 16 hex chars (`@1f4b195aefa10e26`). Drop the `[..16]` slice so the full 64-char Blake3 hex is printed (`@1f4b195aefa10e2690d13c5b98b3d9124d3fbb5c…`), matching what tooling like `--fail-out` records and what the metadata-free workflow actually needs to identify a constant. - `--workers N` flag on `ix check` plumbs through the existing `IX_KERNEL_CHECK_WORKERS` env var that `resolve_kernel_check_workers` in `src/ffi/kernel.rs` reads. Useful for isolating per-worker memory cost (`--workers 1` lets you see the env's static footprint without per-worker overhead) and for capping concurrency in resource-tight contexts. * Mmap .ixe + cache-free LazyConstant for anon mode `lake exe ix check compilemathlib.ixe --anon` on a 3.2 GB env now peaks at ~11 GB RSS, down from ~40 GB; build_anon_work is 50× faster. Three intertwined changes: 1. Memory-map the .ixe (avoids ~3.2 GB heap copy of bytes) - Cargo: add memmap2 = "0.9". - `BytesSource` enum in `src/ix/ixon/lazy.rs` distinguishes heap-resident `Arc<[u8]>` from `(Arc<Mmap>, offset, len)` windows into a memory-mapped file. `LazyConstant::raw_bytes`, `verify_address`, `PartialEq` route through `BytesSource::as_slice` so every existing consumer is mmap-transparent. - `Env::get_anon_mmap(path)` in `src/ix/ixon/serialize.rs` opens the file, mmaps it, and stores Section-2 consts as mmap-backed `LazyConstant`s. Sections 3-4 (names + named) are still parsed transiently to harvest hints, then dropped before return. - `rs_kernel_check_anon` (`src/ffi/kernel.rs`) switches to `get_anon_mmap`; the old `std::fs::read` + `Env::get_anon` heap path is gone for the anon FFI. 2. Strip the persistent `LazyConstant` parsed-Constant cache - `LazyConstant.cache` was `Arc<OnceLock<Arc<Constant>>>` and accumulated forever — for mathlib that meant ~30 GB of parsed `Arc<Expr>` trees pinned in the env across the entire run. - New shape: `cache: Option<Arc<Constant>>`. Populated only by `from_constant` (compile-side, where we already own the parsed value). `from_bytes`/`from_mmap_slice` set `None`, and their `get()` parses fresh on every call without storing. - Re-parse cost is bounded by the existing per-worker dedup: `kenv.consts` already addresses each ingressed constant once per work item, and `clear_releasing_memory()` drops the kenv between items. The kenv is now the only persistent materialization layer. 3. `LazyConstant::peek_variant` for `build_anon_work` - One-byte read of the outer Tag4 head to identify the `ConstantInfo` variant — no body parse, no allocation. - `build_anon_work` now dispatches on `peek_variant()`; only `Muts` blocks trigger `lc.get()` (we need the member list for projection-address enumeration), and that `Arc<Constant>` drops at the end of the match arm. - Previously every constant was fully materialized at startup just to read its variant tag. With ~95% of the env being standalone/projection, the work-enumeration pass now skims the env at near-IO speed. Test updates: - `mmap_slice_roundtrips` (already added with the earlier mmap scaffolding) exercises the mmap window roundtrip. - New `from_bytes_does_not_cache` and `from_constant_clones_share_cache` document the new caching contract. - `peek_variant_*` tests cover every variant + empty-bytes and unknown-flag error paths. - `lazy_sparsity_only_materializes_closure` in `src/ix/ixon/env.rs` reframed to assert BFS-of-closure correctness instead of cache side-effects (`is_materialized()` no longer fires for lazy loads). End-to-end on compilemathlib.ixe (--anon, 32 workers): before: 640658/640658 in 266s, peak RSS ~40 GB after: 640658/640658 in 223s, peak RSS 11.3 GB * Address review findings: correctness + small refactors Code-review pass over the anon-mode / mmap / cache-strip work. This commit lands the highest-value subset — the correctness fixes and the small refactors that prevent future drift — and leaves the larger items (format-version bump, sealed marker trait, AnonEnv audit) for separate follow-ups. Correctness: - Verify per-constant address on load (#1). `LazyConstant::verify_address` existed but was never invoked from `Env::get`, `get_anon`, or `get_anon_mmap`. The env-level merkle root catches missing/extra entries but not byte-tampering of a constant whose key is intact; without this check, corruption surfaced much later inside `LazyConstant::get` with a misleading parse error. Inline the `Address::hash(bytes) == addr` check in each loader's Section-2 loop. Added 3 corruption-detection tests (`env_const_bytes_tampering_*`). This required updating a handful of existing tests that stored constants under fake `Address::hash(b"a")` keys instead of their true content hashes — round-tripping such envs now correctly rejects. Added a `store_canonical(env, c) -> Address` test helper for the canonical pattern, and `*_discriminator(refs, n)` so tests can produce content-distinct constants when the same ref-set would otherwise collide. - Hard-error in `ixon_env_to_decoded` on parse failure (#6). `src/ffi/ixon/env.rs` used `filter_map` to silently drop any const whose bytes failed `LazyConstant::get` — the Lean caller had no signal of the lost entries. Switch to a Result-collecting loop and propagate the first parse error; update both FFI call sites (`rs_de_env`, `rs_de_env_anon`). - Doc fixes (#4, #5). `docs/Ixon.md`'s AssumptionTree section described only `Leaf=0x00` and `Node=0x01`, missing `Padding=0x01` (and the docs' `Node` tag collided with the real `Padding` tag — the real `Node` is `0x02`). Also: the env-section table cell said "opt-tagged merkle root" but the implementation writes a bare 32-byte address. Refactors that prevent future drift: - Canonical projection-address helpers (#9). Four production sites reconstructed `Constant::new(IxonCI::{D,I,R,C}Prj{...}).commit().0` by hand; the anon pipeline silently breaks if any one drifts. Extract `defn_proj_address` / `indc_proj_address` / `recr_proj_address` / `ctor_proj_address` (plus `_constant` variants) in `src/ix/ixon/constant.rs`. Update `compile.rs`, `compile/mutual.rs`, and the anon `*_proj_addr` helpers to call them. - `verify_proj_addr_in_env` helper (#21). `ingress_anon_block` had the same "computed-address not in env" check repeated four times (DPrj/RPrj/IPrj/CPrj). DRY into one helper that produces a consistent error format. UX: - Anon display labels switched to `#<hex>` everywhere (#18). Rust was emitting `@<hex>` in progress/fail-out; Lean's `runCheckAnon` was emitting `#{i}` (result index). Standardize on `#<hex>` so the CLI failure summary is joinable with the fail-out file. This required exposing addresses per result slot to Lean — `rsCheckAnonFFI` now returns `Array (String × Option CheckError)` pairing each result with its content-address hex string. Rust builds the pairs via a new `build_anon_result_array` helper. - Stale "check-ixon" header in FailureLog (#16). One-line rename in the doc comment + `# ix check-ixon failures` writeln to `# ix check failures`. Tests: - `testPrimitivesParity` (PrimAddrs regen-parity test). Catches silent drift between hardcoded primitive addresses in `PrimAddrs::new()` and what `rsCompileEnvFFI` produces from the live Lean primitives. Plumbing: new `PrimAddrs::lean_parity_table()` returns `(lean_name, hex)` pairs; new `rs_prim_addrs_canonical` FFI exposes them to Lean; new test in `BuildPrimitives.lean` iterates `kernelPrimitives`, compares against the hardcoded table, and fails with a printable diff on mismatch (with instructions to regenerate). Skipped: `eagerReduce` — synthetic kernel marker whose PrimAddrs value (`0xff…3`) intentionally diverges from its compiled content hash (which collides with `id`). - 3 new corruption-detection tests in `serialize.rs` exercise the Section-2 verify check for `Env::get`, `Env::get_anon`, `Env::get_anon_mmap`. Verification: `cargo test --lib` 1025 passing (3 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe` 105487/105487 in ~21s, `--anon` 89010/89010 with peak RSS 1.4 GB, `.lake/build/bin/IxTests --ignored rust-kernel-build-primitives` shows both `build primitives dump` and `primitive address parity (PrimAddrs vs live compile)` pass. Out of scope (deferred to follow-ups): - #2 Format version bump (Env::FLAG) - #3 rs_kernel_check_consts_anon still uses Env::get on main - #8 Sealed marker trait for the lazy_anon transmute - #11/#12 AnonEnv audit (vestigial wrapper) - #13, #15, #17, #19, #20, #22-25 Various quality cleanups * Round-2 review fixes: mmap defense + small cleanup Six small items from the round-2 review of the anon-mode work: - N1: stat-at-open defense in `Env::get_anon_mmap` (`src/ix/ixon/serialize.rs`). Capture the file size before mmap; if the kernel's mapped length disagrees (file truncated between open and map) bail with a clear error instead of letting workers SIGBUS deep in `LazyConstant::get`. Truncate-in-place under a live mapping is still undefined per POSIX — documented as a caller contract — but the open-time check catches the common case. - New `get_anon_mmap_survives_file_unlink` test: load via mmap, unlink the path, then materialize both already-touched and not-yet-touched constants. Locks in the inode-retention invariant that the SIGBUS analysis depends on; a future refactor that switched to `mmap_anonymous` or copied bytes into a tmpfile would fail loudly here instead of letting workers SIGBUS in production. - #11: delete `AnonEnv::as_ixon_env_unchecked` (`src/ix/kernel/anon_env.rs`). Dead `pub(crate)` escape hatch marked `#[allow(dead_code)]` — confirmed zero callers and removed. - #13: `debug_assert!` on `OnceLock::set` for both result vectors (`src/ffi/kernel.rs`, meta + anon paths). The previous `let _ = results[idx].set(...)` silently dropped a re-set. If a future `build_*_work` dedup refactor breaks the one-write-per-slot invariant, debug builds now panic with the slot index instead of silently losing results. - #19: `allNames.contains` O(n²) preflight → `Std.HashSet` lookup (`Ix/Cli/CheckCmd.lean`). At mathlib scale (~700k env names × thousands of seed names) the previous linear scan-per-name spent measurable seconds on missing-name preflight alone. - N4: doc imprecision in `src/ix/ixon/lazy.rs` — the cache-policy preamble said the worker `KEnv` is "cleared between work items" but the actual cadence is `clear_every` items (`IX_KERNEL_CHECK_CLEAR_EVERY`, default 1 but tunable). Refined wording so the doc matches the code. Verification: `cargo test --lib` 1026 passing (1 new), `cargo clippy --lib --all-targets` clean, `lake build` clean, `lake exe ix check compileinitstd.ixe --anon` 89010/89010 in 15.2s peak RSS 1.4 GB (regression-clean). Out of scope, separate follow-ups: - #2 Format version bump - #3 rs_kernel_check_consts_anon zkPCC FFI (Env::get → mmap) - #8 Sealed marker trait for the lazy_anon transmute - #10 child_arena closure side-effect (widen MField to tuple) - #12 AnonEnv duplicates bfs_refs / transitive_deps_excl - #14 anon worker bypasses M-generic display helpers - #17 Anon --fail-out docstring claims --consts-file compat - #20 Singleton-unwrap duplicated across compile paths - #22-24 Lean `partial def`, `.get!` in tests, unguarded as-u64 casts - N2/N3 Lean is_materialized() callers + compute_const_size_breakdown - Test gaps: rs_kernel_check_anon integration, concurrent mmap, verify_address-on-mmap-corruption, etc. * rustfmt * Tests: derive RawConst addrs from content (verify_address fix) The `Address::hash(bytes) == addr` defense added in f792102 rejects `RawConst { addr, const }` pairs where `addr` is uncorrelated with `serConstant const`. The Rust-side tests in `serialize.rs` were updated at the time (`store_canonical` helper), but three Lean-side sites still produced mismatched pairs and surface now as: × ∃: Env serialization Lean==Rust × ∃: serde RawEnv with data × ∃: serde RawEnv roundtrip deserialization failed: rs_de_env: Env::get: const at idx 0 bytes hash to 6110b739… but stored under b177ec1b… Three fixes, all derive `addr := Address.blake3 (serConstant c)`: - `Tests/Gen/Ixon.lean::genRawConst` — property-test generator was `RawConst.mk <$> genAddress <*> genConstant` (uncorrelated). Now generates the const first, then derives addr from `serConstant`. - `Tests/FFI/Lifecycle.lean::serdeTests` (`withData` unit case) — was using one `testAddr := Address.blake3 #[1,2,3]` for both the const and unrelated blob/comm slots. Split into `testAddr` (still used for the content-hash-free blob/comm) and `testConstAddr := Address.blake3 (serConstant testConst)`. Added a name entry for the new canonical addr. - `Tests/FFI/Lifecycle.lean::genSerdeRawEnv` — the "pool of addresses" pattern picked const addrs from a random pool that couldn't possibly match content hashes. Restructured: consts derive canonical addrs from content, each gets its own name-table entry appended to the pool-derived entries so the serde pipeline's "all addresses resolvable" invariant still holds. * Regenerate Aiur primitive addresses for singleton-unwrap Compiler's singleton non-inductive Muts unwrap (12630aa) changed the content hashes of 48 primitives (`Nat.rec`, `Nat.add`, `Nat.mul`, …). The Aiur kernel's `Ix/IxVM/Kernel/Primitive.lean` still held the pre-unwrap blake3 bytes, so primitive dispatch missed every affected constant and fell through to structural `Nat.rec` unfolding. Tests like `IxVMPrim.nat_mul_big` then drove millions of Nat.succ unfoldings inside the Aiur trace, OOMing the prover. Addresses regenerated from `PrimAddrs::lean_parity_table()` (which the branch had already updated in `src/ix/kernel/primitive.rs`). * Fix standalone Recr ingress: typ-based inductive lookup `find_matching_block_addr` heuristic in the standalone-Recr branch of `build_convert_inputs_walk` picks the inductive block by matching ctor count to rule count. When multiple in-scope inductives share the same ctor count, it returns the wrong block — the stored rules' `ctor_idx` then points to a sibling's ctor, while `populate_rules` (canonical) derives `ctor_idx` from the right inductive's `ctor_indices`. The mismatch surfaces as `compare_rules`'s `assert_eq!(s_ctor, c_ctor)` panic (e.g. `38 != 40` for `IxVMPrim.nat_land_lit`). Switch the standalone-Recr path to the same typ-based resolution `build_aux_recr_ctor_idxs` already uses for aux Recr blocks: peel `params + motives + minors + indices` foralls of `recr.typ`, take the major's head Ref, look it up in `refs` to get the inductive's address, then resolve `IPrj.block` for the Muts wrapper. Slice ctor positions for the specific member via `extract_member_ctor_idxs`. Also store the resolved Muts `block_addr` in `CKRecr` (was passing the heuristic's wrong address through to `convert_recursor`). --------- Co-authored-by: Arthur Paulino <arthurleonardo.ap@gmail.com> Co-authored-by: Samuel Burnham <45365069+samuelburnham@users.noreply.github.com>

plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed).

* Aiur Bytecode → Rust codegen for the IxVM kernel Adds a new Aiur pipeline stage that translates `Bytecode.Toplevel` into a Rust source module, one `fn aiur_fn_N` per Aiur function. The generated code mirrors `src/aiur/execute.rs`'s QueryRecord side effects exactly: same `function_queries.insert` timing, same cache-hit multiplicity bumps, same memory-queries insertion order, same `bytes{1,2}_queries` updates, same IO sequencing. Per-witness trace hashes match the interpreter byte-for-byte on every const tested (`Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). * `Ix/Aiur/Stages/Codegen.lean`: structured Rust IR (`RustExpr` / `RustStmt` / `RustItem` / `MatchArm` / labeled blocks) plus a formatter. Codegen is a structural walk over the Bytecode AST, no string templating. `EmitM = StateM EmitState` threads `(nextVal, nextLabel)` so per-ValIdx Rust locals and per-MC labels are fresh. * Aiur's `map: Vec<G>` is gone in the generated kernel: every ValIdx becomes a Rust local `__v_{i}: G`. Match-arm bodies snapshot `nextVal` on entry so per-arm allocations don't leak to siblings. `MatchContinue` / `Yield` use labeled-block + `break 'label [G; OUT_SIZE]` to bubble yielded values out and rebind them at outer scope. * Deep Aiur recursion uses `stacker::maybe_grow(64 KiB, 4 MiB, …)` at the top of every generated fn, so the native call stack grows on demand rather than pre-reserving a giant thread stack. Verified against shard 24 of the 64-way `init.ixes` partition, which SEGFAULTs without it. * `ix codegen`: new CLI command. Compiles the IxVM Aiur source, walks the bytecode, writes the generated Rust to a fixed path (`src/ix/aiur_ixvm.rs`). The output path is hard-coded; no override. * `src/ix/aiur_ixvm.rs`: the generated kernel, 743 `aiur_fn_*` + `execute_generated` dispatch. Auto-regenerated; do not edit. `#![cfg_attr(rustfmt, rustfmt::skip)]` + a broad `#![allow(unused_*, non_snake_case, clippy::all)]` since the layout is for the compiler, not humans. * `src/ix/aiur_ixvm_runner.rs`: `execute_ixvm(toplevel, fun_idx, args, io_buffer) -> Result<(QueryRecord, Vec<G>), ExecError>`. Same return shape as `Toplevel::execute`, but routes through `execute_generated`. * `src/aiur/synthesis.rs`: `AiurSystem::prove_ixvm(...)` — same shape as `prove`, but the execute step calls `execute_ixvm`. Verification-compatible (proofs from one path verify under the other). * `src/aiur/execute.rs`: helpers exposed for the codegen'd kernel (`bytes{1,2}_execute` → `pub(crate)`, `unconstrained_big_uint_div_mod_helper` extracted, `CodegenBytes{1,2}{Op,}` aliases re-exported, `QueryRecord::new` → `pub(crate)`, `ExecError::InvalidFunIdx` added). * `src/ffi/aiur/protocol.rs`: two new FFI exports — `rs_aiur_toplevel_execute_ixvm` and `rs_aiur_system_prove_ixvm`. Same wire format as the existing `_execute` / `prove` exports. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.executeIxVM` — mirror of `execute`. * `Ix/Aiur/Protocol.lean`: `AiurSystem.proveIxVM`. * `Ix/Cli/CheckCmd.lean`: `runCompiled` now dispatches through `executeIxVM`. The Rust bytecode interpreter is no longer reachable from `ix check` (the Lean-side `--interp` fallback stays for richer error diagnostics). * `Ix/Cli/ProveCmd.lean`: `proveOne` uses `proveIxVM`. * `Cargo.toml`: `stacker = "0.1"`. For witnesses where execute is the dominant work (single-const checks via `ix check 'X'`), the codegen'd kernel is ~1.6× faster than the bytecode interpreter end-to-end (measured on `Nat.add_comm`, `Vector.append`, `Std.Time.Week.Offset.ofMilliseconds`). Tiny consts are within ~10% (stacker probe overhead amortises). Peak RSS is within a few MB of the interpreter — stacker grows the stack on demand, no giant upfront reservation. For shard-driven `ix check --shard K` runs the speedup is invisible: per-shard wall time is dominated (~92%) by `buildShardCheckEnvWitness` in Lean, NOT by the kernel execute (~8%). Cutting witness construction is the next lever. `Nat.add_comm` proof produced via `proveIxVM` verifies under the existing `AiurSystem::verify`. End-to-end: lake exe ix prove 'Nat.add_comm' → proof addr 0d0ab0f9… lake exe ix verify 0d0ab0f9… → ok * Move shard witness construction to Rust + parallelise Replaces `IxVM.ClaimHarness.buildShardCheckEnvWitness` (Lean side, ~92% of shard wall time on heavy partitions) with a Rust port that builds the `aiur::execute::IOBuffer` directly, without per-byte boxing into Lean `Aiur.G` values. The two hot phases run on rayon: * Closure walk: each owned addr's transitive `Constant.refs` + projection-block traversal runs on its own thread; results are deduped through a `dashmap::DashSet`. * Byte-to-G conversion: per-const `(key, data)` tuples are built in parallel chunks of 256; final IOBuffer assembly (channel arena append + key→`IOKeyInfo` map insert) runs serially because the arena index is monotonic. * `src/ix/aiur_ixvm_witness.rs` (new): `build_shard_check_env_witness` produces `(claim, claim_digest_input, io_buffer)` ready to feed to `execute_ixvm`. Mirrors the 6-channel layout documented in `Ix/IxVM/ClaimHarness.lean` (claim / asm tree / const bytes / Defn hint / blob discriminator / blob raw bytes). * `src/ffi/aiur/protocol.rs`: new FFI `rs_aiur_toplevel_shard_check_ixvm` bundles witness build + `execute_ixvm` into one cross-language trip. Returns the same `(output, ioBuffer, queryCounts)` shape as `rs_aiur_toplevel_execute_ixvm`, so the Lean shim stays drop-in compatible. * `Ix/Aiur/Semantics/BytecodeFfi.lean`: `Bytecode.Toplevel.shardCheckIxVM` wraps the FFI. * `Ix/Cli/CheckCmd.lean`: shard-mode `ix check` (`--ixe + --ixes`) now dispatches through `runShardOwnedNative` → `shardCheckIxVM`, bypassing the Lean witness builder. Single- shard and whole-partition paths share the fast route; the legacy `runShardCheckManifest` / `runShardCheckAll` callbacks stay live for `--interp` only. Shard 26 of the 64-way `init.ixes` partition, on the same host: | Variant | total | speedup | | --- | --: | --: | | Lean witness + bytecode-interp (baseline)| 1029 s | 1.0× | | Lean witness + codegen kernel | 1015 s | 1.01× | | Rust witness (serial) + codegen | 101 s | 9.95× | | Rust witness (parallel) + codegen | 80 s | 12.9× | The serial Rust witness collapses the 935 s of Lean per-byte boxing into ~23 s; rayon hides that 23 s under the codegen-kernel execute (~78 s) so witness build effectively becomes free at the shard level. FFT cost matches the bytecode interpreter exactly (107_006_963_281), so the QueryRecord layout is bit-identical across all four variants. * Closure walk uses single-source BFS with the global visited set shared via `DashSet::contains` checks before pushing to the per-thread stack — avoids redundant work without a per-owned union step. * Per-channel ordering: each chunk's partial `ChannelEntries` feeds the serial fold in iteration order, so channel arena contents match the Lean side's exactly. Test still relies on trace-hash parity from the earlier codegen commit, which exercised the same path through the bytecode interpreter and the codegen kernel. * Coverage check is currently skipped on the IxVM-native whole-partition path (`runShardManifestAllNative`); legacy `runShardCheckAll` still does it for `--interp`. Re-enable separately if needed. * Codegen: value-returning byte ops, dead-fold the unconstrained plumbing, unchecked cache-hit array copy Three targeted cuts to the generated kernel (`src/ix/aiur_ixvm.rs`); all preserve QueryRecord parity (shard 26 FFT cost unchanged at 107_006_963_281). Every `U8*` op previously emitted a `Vec<G>` scratch (~245 sites): let __b2_out: [G; 1] = { let mut __scratch: Vec<G> = vec![__v_i, __v_j]; if unconstrained { __scratch.extend(vec![Bytes2::xor(...)]); } else { bytes2_execute(0, 1, &Bytes2Op::Xor, &mut __scratch, record); } let __arr: [G; 1] = __scratch[2..].try_into().unwrap(); __arr }; Net cost: a 2-element `vec![]` alloc + a `__scratch.extend(vec![..])` (another small alloc inside `Bytes2::execute`'s returned Vec) + a slice→array `try_into().unwrap()`. Added per-op `bytes{1,2}_*_value(args..., record) -> G | (G,G) | (G,G,G) | [G; 8]` in `src/aiur/execute.rs` — each bumps the corresponding `bytes{1,2}_queries.bump_*` and returns the gadget output by value. The pure (`Bytes{1,2}::*`) helpers stay the unconstrained shortcut. Codegen now emits: let __v_n: G = if unconstrained { Bytes2::xor(&__v_i, &__v_j) } else { bytes2_xor_value(__v_i, __v_j, record) }; Zero `Vec<G>` allocation per byte op. `U8Add`/`U8Sub` also gain because `bytes2_add_value` runs the `Bytes2::add` gadget once (returning the full `(low, carry)`) instead of the interpreter's two `Bytes2::add` calls (one for the carry, one for the low push). To make `bump_*` callable from `execute.rs`, all `Bytes1Queries` / `Bytes2Queries::bump_*` methods are now `pub(crate)`. No behaviour change. The `Op::Call` cache-check emitted `unconstrained || OP_UN` and `!unconstrained && !OP_UN` even when the static `OP_UN` was `false`, which is the common case. Codegen now emits the folded shape directly: let __cu = unconstrained; // was: unconstrained || false if !unconstrained { *result.multiplicity += G::ONE; } // was: !unconstrained && !false When `OP_UN == true`, the callee always runs unconstrained AND the multiplicity bump is always suppressed; codegen folds to `let __cu = true;` and elides the bump branch entirely. LLVM was already folding these, so this is mostly source-cleanliness and a small frontend win. The cache-hit branch read `result.output` (an `&[G]`) and converted it back to `[G; OUT_N]` via `.try_into().unwrap()`. The length is statically `OUT_N` — we control the producer (the matching `aiur_fn_{callee}::Ctrl::Return` inserts an `[G; OUT_N]`-typed array into the same slot). The runtime bounds-check + Result discharge is dead work. Codegen now emits: let __ret: [G; OUT_N] = unsafe { *(result.output.as_ptr() as *const [G; OUT_N]) }; Sound: same-fn slot, fixed `OUT_N`, no aliasing. | Variant | mean | | --- | --: | | Rust witness (parallel) + codegen | 80 s | | + value-helper byte ops (#1) | 64 s | | + folded `__cu` + unsafe cache copy (#2+#3) | 62 s | The bulk of the win is #1 (#1 alone: ~−18%). #2+#3 are a further ~3% — mostly noise / Rust-frontend cleanup since LLVM already folds the constant `false` ops. * `src/aiur/execute.rs`: 12 value-returning byte helpers added (`bytes1_bit_decompose_value`, `bytes1_shift_left_value`, `bytes1_shift_right_value`, `bytes2_{xor,and,or,less_than,mul, chain_rotr7,chain_rotr4,add,sub}_value`). * `src/aiur/gadgets/bytes1.rs`, `src/aiur/gadgets/bytes2.rs`: all `bump_*` upgraded to `pub(crate)` so the value helpers can call them. * `Ix/Aiur/Stages/Codegen.lean`: `emitU8Bytes1` / `emitU8Bytes2` / `emitU8Add` / `emitU8Sub` rewritten to call the per-op value helpers — no scratch Vec, no slice→array conversion. `emitCall` constant-folds `opUn = false` and emits the unsafe cache-hit copy. Prelude imports updated to bring the new helpers into scope. * `src/ix/aiur_ixvm.rs`: regenerated. 245 `Vec<G>` scratch sites → 11 (only `unconstrainedBigUintDivMod`'s scratch left); 3324 `unconstrained || false` → 0; 3000+ `result.output.try_into() .unwrap()` → 0. * `Ix/Cli/CheckCmd.lean`: incidental cleanup of probe-only timing prints in `runShardOwnedNative` (added during measurement, no longer needed). * Wire Rust witness fast path through every check/prove entry point Per-claim mode (a `Claim.check addr none`) builds a closure rooted at the target address. That closure can be the whole environment when the target is a heavily-shared root constant. The Lean witness builder paid per-byte boxing into `Aiur.G` for every byte in that closure — the same cost we already eliminated for shard mode. # Surface Five new FFIs, all bundling witness build + execute_ixvm (and STARK prove where applicable) into one cross-language trip so the `IOBuffer` never crosses the boundary mid-pipeline: * `rs_aiur_toplevel_check_addr_ixvm` — `Bytecode.Toplevel.checkAddrIxVM (toplevel) (funIdx) (ixePath) (addrBytes)`: per-claim check; builds witness for `Claim.check addr none` via `build_claim_check_witness`, runs `execute_ixvm`. Same return shape as `shardCheckIxVM`. * `rs_aiur_toplevel_check_env_bytes_ixvm` — same as above but takes the env as a serialized byte blob (`Ixon.serEnv`) instead of a `.ixe` path. Used by the compiled-Lean-env code path (`ix check NAME` without `--ixe`), where the env is built in Lean memory. * `rs_aiur_system_prove_addr_ixvm` — `AiurSystem.proveAddrIxVM (...)`: per-claim prove (witness + execute + STARK prove all in Rust). * `rs_aiur_system_prove_env_bytes_ixvm` — bytes-blob counterpart. * `rs_aiur_system_shard_prove_ixvm` — `AiurSystem.shardProveIxVM (...)`: per-shard prove end-to-end in Rust. `build_claim_check_witness` lives next to `build_shard_check_env_witness` in `src/ix/aiur_ixvm_witness.rs` and uses the same parallel closure walk + parallel byte→G conversion. The bytes-blob FFIs additionally harvest `anon_hints` from each `Def` named entry after `Env::get` — `Env::get` (full form, used by the bytes-blob path) doesn't populate `env.anon_hints` the way `Env::get_anon` does, but the kernel's `verify_claim` reads ch 3 (Defn reducibility hints) so the hints must end up in the env. This mirrors the harvest pass already inside `get_anon` at `src/ix/ixon/serialize.rs:1683`. # Wiring `Ix.Cli.CheckCmd.WitnessSource`: a three-arm discriminated union threaded through `forEachClaim` (the shared check/prove driver): * `.native ixePath addr` — `.ixe`-backed env, Rust mmap. * `.nativeBytes envBytes addr` — Lean-memory env serialized to bytes, Rust decodes via `Env::get`. * `.lean witness` — pre-built `ClaimWitness` (fallback). `runCompiled` and `proveOne` dispatch on the source. # Coverage | Command | Path | |---------------------------------------|-----------------------------------------------| | `ix check NAME --ixe` | **`checkAddrIxVM`** (new) | | `ix check NAME` (no `--ixe`) | **`checkEnvBytesIxVM`** (new) | | `ix check --claim hex --ixe` | `checkAddrIxVM` for `check addr none` | | `ix check --ixes --shard K` | `shardCheckIxVM` (existed) | | `ix check --ixes` | `shardCheckIxVM` (existed) | | `ix prove NAME --ixe` | **`proveAddrIxVM`** (new) | | `ix prove NAME` (no `--ixe`) | **`proveEnvBytesIxVM`** (new) | | `ix prove --ixes --shard K` | **`shardProveIxVM`** (new) | | `ix prove --ixes` | **`shardProveIxVM`** loop (new) | | `Benchmarks/Typecheck.lean` Phase 1 | `checkAddrIxVM` / `executeIxVM` | | `Benchmarks/Typecheck.lean` Phase 2 | `proveAddrIxVM` / `proveIxVM` | | `Benchmarks/IxVM.lean` | `proveIxVM` (was `prove`) | `--interp` route is preserved: it materialises any `WitnessSource` back into a `ClaimWitness` before driving the Aiur source interpreter. `--claim <hex>` over non-`check addr none` variants (`eval` / `reveal` / `contains` / `checkEnv-with-asm`) still uses the Lean witness builder. # Sanity * `ix check Nat.add_comm --ixe init.ixe` (warm): 3.1 s wall, FFT = 23_603_449. * `ix check Nat.add_comm` (compiled env via bytes blob, warm): 2.4 s wall, FFT = 23_603_449. # Known overheads (tracked for a follow-up redesign) * **Per-claim env re-parse on `--ixe + many names`**: each FFI call re-runs `Env::get_anon_mmap` on the same path. Mathlib-scale iteration pays the lazy-index build N times instead of once. * **`--interp + .nativeBytes` round-trip**: Lean serializes the compiled env then `materialise` deserializes it back, just to feed `runInterp`. * **`runShardProveNative` claim reconstruct**: redoes the closure walk + canonical `AssumptionTree` build Lean-side after `shardProveIxVM` already computed the same claim internally. A future `EnvHandle`-based API would close all three: build the env once into a Rust-owned handle, pass the handle to every per-claim/per-shard FFI, and have prove FFIs return the serialized claim bytes. * EnvHandle: Rust-owned env shared across per-claim / per-shard FFI calls Before: every per-claim and per-shard FFI re-parsed the Ixon env (`Env::get_anon_mmap` per call). On `--ixe + many names` / all-shards prove, this is the dominant overhead for iteration-heavy workflows — O(num_consts) lazy-index build × N targets. After: the env lives once per CLI invocation in a Rust-owned `EnvHandle`. Lean holds an opaque `Aiur.EnvHandle` reference and threads `@& EnvHandle` through every per-target FFI call. The env is parsed exactly once at handle construction; downstream calls share it. # Surface Five new FFIs collapse the previous six per-call variants (`checkAddrIxVM`, `checkEnvBytesIxVM`, `shardCheckIxVM`, `proveAddrIxVM`, `proveEnvBytesIxVM`, `shardProveIxVM` — all deleted): * `rs_aiur_env_handle_from_ixe(path) → LeanExternal<EnvHandle>`: mmap-load via `Env::get_anon_mmap`. Anon parser already harvests `anon_hints`; no post-pass. * `rs_aiur_env_handle_from_bytes(blob) → LeanExternal<EnvHandle>`: decode `Ixon.serEnv`-shape blob via `Env::get` + harvest `anon_hints` from each `Def` named entry. Used by the compiled-Lean-env path. * `rs_aiur_toplevel_check_addr_with_env(toplevel, fun_idx, handle, addr_bytes)`: per-claim check. Reuses handle's parsed env. * `rs_aiur_toplevel_shard_check_with_env(toplevel, fun_idx, handle, owned_blob)`: per-shard check. * `rs_aiur_system_prove_addr_with_env(system, fri, fun_idx, handle, addr_bytes) → (claim_bytes, proof, ioBuffer)`: per-claim prove. Rust serializes the reconstructed `Ix.Claim` via `ixon::Claim::put` so Lean can deserialize via `Ixon.runGet Ix.Claim.get` directly — no closure walk + canonical `AssumptionTree` recomputation Lean-side. * `rs_aiur_system_shard_prove_with_env(system, fri, fun_idx, handle, owned_blob) → (claim_bytes, proof, ioBuffer)`: per-shard prove. # Lean * `Aiur.EnvHandle` opaque type with `fromIxe` / `fromBytes`. * `Aiur.Bytecode.Toplevel.checkAddrWithEnv` / `shardCheckWithEnv`, `Aiur.AiurSystem.proveAddrWithEnv` / `shardProveWithEnv`. * `Ix.Cli.CheckCmd.Target` replaces `WitnessSource`: ```lean inductive Target where | addr (a : Address) -- Claim.check addr none | shard (owned : Array Address) -- Claim.checkEnv | leanW (w : ClaimWitness) -- --interp, --claim non-check ``` `runCompiled` / `proveOne` take `(envHandle?, target)`. The envHandle is `none` only for `.leanW` (`--interp` legacy path). * `runShardOwnedNative` and the manifest-driver helpers take the envHandle as a parameter. Single-shard mode builds it once; all-shards mode reuses the same handle across every shard's FFI call (eliminates per-shard re-mmap). * `runShardProveNative` deserializes the wire claim bytes via `Ixon.runGet Ix.Claim.get` instead of re-running `shardCheckEnvClaim`. # Benchmarks `Benchmarks/Typecheck.lean` builds the envHandle once before Phase 1, reuses it across Phase 1 (execute) + Phase 2 (prove). Both phases now go through `checkAddrWithEnv` / `proveAddrWithEnv` on the full-closure path. `--subject-only` still uses Lean `buildVerifyConst` + `executeIxVM` / `proveIxVM` (witnesses intentionally small there). # New crate `crates/ix/src/env_handle.rs` — the `EnvHandle` struct + constructors live in the existing `ix` crate next to the witness builder and codegen runner. # Sanity * Multi-target check (warm): `ix check --ixe init.ixe Nat.add_comm Nat.add` → 3.3 s wall, single envHandle shared across both targets. * Shard 26 check (warm): `ix check --ixe init.ixe --ixes init.ixes --shard 26` → ~78 s wall, FFT = 107_006_963_281 (parity preserved). # Coverage | Command | Path | |--------------------------------------|-----------------------------------| | `ix check NAME --ixe` | `checkAddrWithEnv` | | `ix check NAME` (no `--ixe`) | `checkAddrWithEnv` + fromBytes | | `ix check --claim hex --ixe` | `checkAddrWithEnv` for check-none | | `ix check --ixes --shard K` | `shardCheckWithEnv` | | `ix check --ixes` | `shardCheckWithEnv` × all shards | | `ix prove NAME --ixe` | `proveAddrWithEnv` | | `ix prove --ixes --shard K` | `shardProveWithEnv` | | `ix prove --ixes` | `shardProveWithEnv` × all shards | | `Benchmarks/Typecheck` Phase 1+2 | `checkAddrWithEnv` / `proveAddrWithEnv` | Only `--interp` and `--claim hex` over a non-`check addr none` persisted claim still build a Lean `ClaimWitness`; both go via the `.leanW` target arm. * ix codegen: add --check flag for CI; gate stale generated kernel `ix codegen --check` compares the emitted Rust source against the on-disk `crates/ix/src/aiur_ixvm.rs` and exits 0 if identical, 1 otherwise. No write side effect. ~2 s warm; fast enough to gate on every PR. Wired into the `lean-test` CI job so a forgotten regen on a kernel-touching PR fails CI instead of merging stale generated code that drifts from the Bytecode → Rust emitter. * ix check: --interp {source|bytecode} to bypass codegen kernel Two interpreter modes behind a single `--interp` flag: * `--interp source`: Aiur source interpreter (`Aiur.runFunction` over the source-level `Decls`). Richer per-step error diagnostics. * `--interp bytecode`: generic Aiur bytecode interpreter (`Bytecode.Toplevel.execute`, `rs_aiur_toplevel_execute` route). Skips the `ix codegen` + `cargo build --release` cycle needed after editing `Ix/IxVM/*.lean` — the bytecode is rebuilt Lean-side at exe load. Slower per-check than the codegen kernel; ideal for tight iteration on the IxVM source. * omit the flag entirely for the native codegen kernel (default). Invalid values (e.g. `--interp foo`) fail fast with a clear error. # Plumbing The `check_addr_with_env` and `shard_check_with_env` FFIs take a `use_bytecode: bool` and dispatch via a shared `dispatch_execute` helper. `--interp bytecode` sets it to `true`. The `.leanW` arm of `runCompiled` also picks between `Bytecode.Toplevel.execute` and `executeIxVM` based on the same flag. The `--interp source` path requires a Lean `ClaimWitness`. The existing driver had switched to `.addr`/`.shard` targets against the Rust-owned `EnvHandle`; source-interp had no way to materialise those. `forEachClaim` now takes a `forceLeanWitness : Bool` — when true, it builds a Lean witness for every target (via `mkWitness` / `loadIxonEnv` for the compiled-Lean-env path) and passes `.leanW` so `runInterp` can consume it directly. # Sanity * `ix check Std.Time.Week.Offset.ofMilliseconds` (warm): 9.3 s wall, FFT = 12_430_516_949 (codegen kernel). * `ix check --interp bytecode Std.Time.Week.Offset.ofMilliseconds` (warm): 14.6 s wall, FFT = 12_430_516_949 (bytecode interp). * `ix check --interp source Eq` (warm): 6.3 s wall, output `Eq: ()`. * `ix check --interp garbage Nat.add_comm`: exits 1 with clear error. * IxVM codegen: regen + rebase-fixups after KLevel port Aligns the ap/codegen-ixvm-native stack with main's KLevel pointer port (b84a500) and re-lands clippy-clean under the workspace lint set: - Regen crates/ix/src/aiur_ixvm.rs from Aiur source (KLevel = &KLevelNode changes the generated fn shapes; ~26k lines net churn). - Extend the generated file's #![allow(...)] header (via Ix/Aiur/Stages/Codegen.lean) with clippy::ptr_as_ptr, clippy::match_same_arms, and clippy::large_types_passed_by_value — three lints the codegen's straight-line style trips deterministically on every regen and that clippy::all doesn't cover. - rustfmt reflow in crates/aiur/src/execute.rs and a handful of ix / ffi files that CI now enforces via the ap/aiur-aux-recursor-parity toolchain. - Fix 4 real clippy warnings uncovered under the workspace lint set: * env_handle.rs: clone-on-Copy → deref (ReducibilityHints is Copy). * aiur_ixvm_witness.rs: collapse if-let-if, drop needless continue, slice::from_ref instead of &[x.clone()]. * aiur_ixvm_runner.rs: keep execute_ixvm's Vec<G> arg (required by AiurSystem::prove_ixvm's fn-pointer bound) + local #[allow( clippy::needless_pass_by_value)] with a comment explaining why. * ffi/aiur/protocol.rs: unqualify aiur::G/IOBuffer/QueryRecord after the workspace lint added unused_qualifications; is_multiple_of() over % 0. Verified: cargo clippy --workspace --all-targets --all-features --D warnings clean; lake test -- --ignored ixvm passes. * ix-check + ixvm tests: coverage gate, --interp source fix, codegen parity + rename ix crate to ixvm-codegen Four review fixes on the codegen-ixvm-native stack. - **Coverage gate on \`--ixes\` (all-shards, native path).** \`runShardManifestAllNative\` now calls \`shardsCover\` before running any shard, matching the pre-refactor \`runShardCheckAll\` behavior. Without this, a stale/truncated \`.ixes\` manifest makes \`ix check --ixes\` exit 0 while some env constants are never checked by any shard — the mundane trigger being: edit a definition, rebuild the \`.ixe\`, forget to re-run \`ix shard\`, and the constants that go unchecked are exactly the ones just edited. Single-shard \`--shard K\` path is unchanged (consistent with pre-refactor). \`shardsCover\` moved above \`runShardManifestAllNative\` to satisfy forward-decl ordering. - **\`ix check --ixe … --claim <hex> --interp source\` regression.** The \`claimHex\` arm in \`forEachClaim\` ignored \`forceLeanWitness\` and always mapped \`.check addr none\` claims to \`Target.addr\`. Under \`--interp source\`, \`runInterp\` then rejects \`.addr\` with "\`--interp requires a Lean witness; addr/shard targets unreachable here\`". Now mirrors the sibling name-based arms and routes through the Lean witness path when \`forceLeanWitness\` is on. Only bit the common claim shape. - **Codegen ↔ bytecode parity CI test.** New \`runParityCase\` + \`parityCases\` in \`Tests/Ix/IxVM.lean\` wire every constant already listed in \`kernelCheckEntries\` plus \`kernel_unit_tests\` through BOTH \`Toplevel.execute\` (bytecode) and \`Toplevel.executeIxVM\` (codegen'd Rust), and diff the returned \`(output, IOBuffer, QueryCounts)\` triples. Turns "generated kernel ≡ interpreter on QueryRecord" from reviewed-by-hand into checked-by-CI. 129 assertions run per suite invocation (43 constants × output + IOBuffer + QueryCount), all pass on this branch head. - **Rename \`crates/ix\` → \`crates/ixvm-codegen\`.** The old \`ix\` crate held only the codegen'd IxVM kernel + its runner + witness helpers. \`ix-kernel\` is the hand-written Rust ix typechecker; \`ix-common\` / \`ix-compile\` are Ix-level infrastructure. \`ixvm-codegen\` cleanly names what this crate holds. Path updates: workspace \`members\` and \`workspace.dependencies\`, ffi crate dep + \`use\` sites, Lean \`codegenOutPath\` in \`Ix/Cli/CodegenCmd.lean\`, doc reference in \`Ix/Aiur/Semantics/BytecodeFfi.lean\`. \`ix codegen\` now writes to \`crates/ixvm-codegen/src/aiur_ixvm.rs\`. Verified: \`cargo check --workspace\` + \`lake test -- --ignored ixvm\` both exit 0 post-rename; parity + pinned FFT tests all pass. * cargo-deny: allow ar_archive_writer's Apache-2.0 WITH LLVM-exception Post-rename, ixvm-codegen pulls stacker → psm → ar_archive_writer (a build-time transitive). ar_archive_writer is licensed under the LLVM-exception variant of Apache-2.0 (standard for LLVM tooling; no additional runtime obligation for downstream users), which isn't on the workspace allow list. Add a per-crate exception scoped to ar_archive_writer so we don't blanket-allow the license family across the tree.

feat: Rust crate
ddd0c78