perf(ix): lazy, zero-copy anon env load for the Aiur check path by samuelburnham · Pull Request #445 · argumentcomputer/ix

samuelburnham · 2026-06-16T19:28:55Z

ix check --ixe deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load.

Make the Lean Ixon.Env constants lazy and load them without copying:

LazyConstant is now an offset window (buf, off, len) into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized.
New rs_de_env_lazy FFI: Env::parse_lazy_index walks the env once (reusing the existing parser, so every metadata variant incl. CallSite is handled) and returns per-const (addr, offset, len) windows into the buffer Lean passed in, plus name -> addr and per-Defn reducibility hints. No constant body is parsed or copied at load time. deEnvAnon reconstructs an Env of byte-window LazyConstants over that same buffer.
This is anon-aligned with the circuit: binder metadata (the ExprMetaArena) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-Defn hint. The CLI's name -> addr map is kept so targets can still be named.
Pure-Lean getEnv/deEnv/rsDeEnv keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (CheckCmd) switches to deEnvAnon.

Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships rawBytes windows directly), shard partitioning, decompile, and the compile/commit env builders.

Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

Builds on the Rust kernel lazy deserialization from #415

arthurpaulino

Nice!
The "perf(aiur)" tripped me up a bit, but the patch looks good.

Merge the Ix-compiler benchmark and a new Aiur execute/prove benchmark into one workflow (aiur-bench.yml). Per library env it runs 'ix compile' to a .ixe (compile-throughput metrics) then STARK-checks selected constants over that same .ixe via bench-typecheck (execute + prove metrics), combined into one bencher report. bench-typecheck loads the env lazily (Ixon.deEnvAnon, #445) so Mathlib statements check at ~10GB peak RSS instead of the ~100GB eager load. Constants are inline in the workflow matrix. Replaces compile.yml.

`ix check --ixe` deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load. Make the Lean `Ixon.Env` constants lazy and load them without copying: - `LazyConstant` is now an offset window `(buf, off, len)` into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized. - New `rs_de_env_lazy` FFI: `Env::parse_lazy_index` walks the env once (reusing the existing parser, so every metadata variant incl. `CallSite` is handled) and returns per-const `(addr, offset, len)` windows into the buffer Lean passed in, plus `name -> addr` and per-`Defn` reducibility hints. No constant body is parsed or copied at load time. `deEnvAnon` reconstructs an `Env` of byte-window `LazyConstant`s over that same buffer. - This is anon-aligned with the circuit: binder metadata (the `ExprMetaArena`) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-`Defn` hint. The CLI's `name -> addr` map is kept so targets can still be named. - Pure-Lean `getEnv`/`deEnv`/`rsDeEnv` keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (`CheckCmd`) switches to `deEnvAnon`. Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships `rawBytes` windows directly), shard partitioning, decompile, and the compile/commit env builders. Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s, and the lazy run additionally typechecks the target rather than only resolving its address. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

samuelburnham · 2026-06-16T19:28:55Z

ix check --ixe deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load.

Make the Lean Ixon.Env constants lazy and load them without copying:

LazyConstant is now an offset window (buf, off, len) into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized.
New rs_de_env_lazy FFI: Env::parse_lazy_index walks the env once (reusing the existing parser, so every metadata variant incl. CallSite is handled) and returns per-const (addr, offset, len) windows into the buffer Lean passed in, plus name -> addr and per-Defn reducibility hints. No constant body is parsed or copied at load time. deEnvAnon reconstructs an Env of byte-window LazyConstants over that same buffer.
This is anon-aligned with the circuit: binder metadata (the ExprMetaArena) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-Defn hint. The CLI's name -> addr map is kept so targets can still be named.
Pure-Lean getEnv/deEnv/rsDeEnv keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (CheckCmd) switches to deEnvAnon.

Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships rawBytes windows directly), shard partitioning, decompile, and the compile/commit env builders.

Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

Builds on the Rust kernel lazy deserialization from #415

arthurpaulino

Nice!
The "perf(aiur)" tripped me up a bit, but the patch looks good.

Merge the Ix-compiler benchmark and a new Aiur execute/prove benchmark into one workflow (aiur-bench.yml). Per library env it runs 'ix compile' to a .ixe (compile-throughput metrics) then STARK-checks selected constants over that same .ixe via bench-typecheck (execute + prove metrics), combined into one bencher report. bench-typecheck loads the env lazily (Ixon.deEnvAnon, #445) so Mathlib statements check at ~10GB peak RSS instead of the ~100GB eager load. Constants are inline in the workflow matrix. Replaces compile.yml.

`ix check --ixe` deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load. Make the Lean `Ixon.Env` constants lazy and load them without copying: - `LazyConstant` is now an offset window `(buf, off, len)` into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized. - New `rs_de_env_lazy` FFI: `Env::parse_lazy_index` walks the env once (reusing the existing parser, so every metadata variant incl. `CallSite` is handled) and returns per-const `(addr, offset, len)` windows into the buffer Lean passed in, plus `name -> addr` and per-`Defn` reducibility hints. No constant body is parsed or copied at load time. `deEnvAnon` reconstructs an `Env` of byte-window `LazyConstant`s over that same buffer. - This is anon-aligned with the circuit: binder metadata (the `ExprMetaArena`) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-`Defn` hint. The CLI's `name -> addr` map is kept so targets can still be named. - Pure-Lean `getEnv`/`deEnv`/`rsDeEnv` keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (`CheckCmd`) switches to `deEnvAnon`. Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships `rawBytes` windows directly), shard partitioning, decompile, and the compile/commit env builders. Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s, and the lazy run additionally typechecks the target rather than only resolving its address. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

samuelburnham · 2026-06-16T19:28:55Z

ix check --ixe deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load.

Make the Lean Ixon.Env constants lazy and load them without copying:

LazyConstant is now an offset window (buf, off, len) into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized.
New rs_de_env_lazy FFI: Env::parse_lazy_index walks the env once (reusing the existing parser, so every metadata variant incl. CallSite is handled) and returns per-const (addr, offset, len) windows into the buffer Lean passed in, plus name -> addr and per-Defn reducibility hints. No constant body is parsed or copied at load time. deEnvAnon reconstructs an Env of byte-window LazyConstants over that same buffer.
This is anon-aligned with the circuit: binder metadata (the ExprMetaArena) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-Defn hint. The CLI's name -> addr map is kept so targets can still be named.
Pure-Lean getEnv/deEnv/rsDeEnv keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (CheckCmd) switches to deEnvAnon.

Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships rawBytes windows directly), shard partitioning, decompile, and the compile/commit env builders.

Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

Builds on the Rust kernel lazy deserialization from #415

arthurpaulino

Nice!
The "perf(aiur)" tripped me up a bit, but the patch looks good.

Merge the Ix-compiler benchmark and a new Aiur execute/prove benchmark into one workflow (aiur-bench.yml). Per library env it runs 'ix compile' to a .ixe (compile-throughput metrics) then STARK-checks selected constants over that same .ixe via bench-typecheck (execute + prove metrics), combined into one bencher report. bench-typecheck loads the env lazily (Ixon.deEnvAnon, #445) so Mathlib statements check at ~10GB peak RSS instead of the ~100GB eager load. Constants are inline in the workflow matrix. Replaces compile.yml.

`ix check --ixe` deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load. Make the Lean `Ixon.Env` constants lazy and load them without copying: - `LazyConstant` is now an offset window `(buf, off, len)` into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized. - New `rs_de_env_lazy` FFI: `Env::parse_lazy_index` walks the env once (reusing the existing parser, so every metadata variant incl. `CallSite` is handled) and returns per-const `(addr, offset, len)` windows into the buffer Lean passed in, plus `name -> addr` and per-`Defn` reducibility hints. No constant body is parsed or copied at load time. `deEnvAnon` reconstructs an `Env` of byte-window `LazyConstant`s over that same buffer. - This is anon-aligned with the circuit: binder metadata (the `ExprMetaArena`) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-`Defn` hint. The CLI's `name -> addr` map is kept so targets can still be named. - Pure-Lean `getEnv`/`deEnv`/`rsDeEnv` keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (`CheckCmd`) switches to `deEnvAnon`. Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships `rawBytes` windows directly), shard partitioning, decompile, and the compile/commit env builders. Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s, and the lazy run additionally typechecks the target rather than only resolving its address. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

samuelburnham · 2026-06-16T19:28:55Z

ix check --ixe deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load.

Make the Lean Ixon.Env constants lazy and load them without copying:

LazyConstant is now an offset window (buf, off, len) into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized.
New rs_de_env_lazy FFI: Env::parse_lazy_index walks the env once (reusing the existing parser, so every metadata variant incl. CallSite is handled) and returns per-const (addr, offset, len) windows into the buffer Lean passed in, plus name -> addr and per-Defn reducibility hints. No constant body is parsed or copied at load time. deEnvAnon reconstructs an Env of byte-window LazyConstants over that same buffer.
This is anon-aligned with the circuit: binder metadata (the ExprMetaArena) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-Defn hint. The CLI's name -> addr map is kept so targets can still be named.
Pure-Lean getEnv/deEnv/rsDeEnv keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (CheckCmd) switches to deEnvAnon.

Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships rawBytes windows directly), shard partitioning, decompile, and the compile/commit env builders.

Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

Builds on the Rust kernel lazy deserialization from #415

arthurpaulino

Nice!
The "perf(aiur)" tripped me up a bit, but the patch looks good.

Merge the Ix-compiler benchmark and a new Aiur execute/prove benchmark into one workflow (aiur-bench.yml). Per library env it runs 'ix compile' to a .ixe (compile-throughput metrics) then STARK-checks selected constants over that same .ixe via bench-typecheck (execute + prove metrics), combined into one bencher report. bench-typecheck loads the env lazily (Ixon.deEnvAnon, #445) so Mathlib statements check at ~10GB peak RSS instead of the ~100GB eager load. Constants are inline in the workflow matrix. Replaces compile.yml.

`ix check --ixe` deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load. Make the Lean `Ixon.Env` constants lazy and load them without copying: - `LazyConstant` is now an offset window `(buf, off, len)` into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized. - New `rs_de_env_lazy` FFI: `Env::parse_lazy_index` walks the env once (reusing the existing parser, so every metadata variant incl. `CallSite` is handled) and returns per-const `(addr, offset, len)` windows into the buffer Lean passed in, plus `name -> addr` and per-`Defn` reducibility hints. No constant body is parsed or copied at load time. `deEnvAnon` reconstructs an `Env` of byte-window `LazyConstant`s over that same buffer. - This is anon-aligned with the circuit: binder metadata (the `ExprMetaArena`) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-`Defn` hint. The CLI's `name -> addr` map is kept so targets can still be named. - Pure-Lean `getEnv`/`deEnv`/`rsDeEnv` keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (`CheckCmd`) switches to `deEnvAnon`. Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships `rawBytes` windows directly), shard partitioning, decompile, and the compile/commit env builders. Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s, and the lazy run additionally typechecks the target rather than only resolving its address. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

samuelburnham · 2026-06-16T19:28:55Z

ix check --ixe deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load.

Make the Lean Ixon.Env constants lazy and load them without copying:

LazyConstant is now an offset window (buf, off, len) into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized.
New rs_de_env_lazy FFI: Env::parse_lazy_index walks the env once (reusing the existing parser, so every metadata variant incl. CallSite is handled) and returns per-const (addr, offset, len) windows into the buffer Lean passed in, plus name -> addr and per-Defn reducibility hints. No constant body is parsed or copied at load time. deEnvAnon reconstructs an Env of byte-window LazyConstants over that same buffer.
This is anon-aligned with the circuit: binder metadata (the ExprMetaArena) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-Defn hint. The CLI's name -> addr map is kept so targets can still be named.
Pure-Lean getEnv/deEnv/rsDeEnv keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (CheckCmd) switches to deEnvAnon.

Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships rawBytes windows directly), shard partitioning, decompile, and the compile/commit env builders.

Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

Builds on the Rust kernel lazy deserialization from #415

arthurpaulino

Nice!
The "perf(aiur)" tripped me up a bit, but the patch looks good.

Merge the Ix-compiler benchmark and a new Aiur execute/prove benchmark into one workflow (aiur-bench.yml). Per library env it runs 'ix compile' to a .ixe (compile-throughput metrics) then STARK-checks selected constants over that same .ixe via bench-typecheck (execute + prove metrics), combined into one bencher report. bench-typecheck loads the env lazily (Ixon.deEnvAnon, #445) so Mathlib statements check at ~10GB peak RSS instead of the ~100GB eager load. Constants are inline in the workflow matrix. Replaces compile.yml.

`ix check --ixe` deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load. Make the Lean `Ixon.Env` constants lazy and load them without copying: - `LazyConstant` is now an offset window `(buf, off, len)` into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized. - New `rs_de_env_lazy` FFI: `Env::parse_lazy_index` walks the env once (reusing the existing parser, so every metadata variant incl. `CallSite` is handled) and returns per-const `(addr, offset, len)` windows into the buffer Lean passed in, plus `name -> addr` and per-`Defn` reducibility hints. No constant body is parsed or copied at load time. `deEnvAnon` reconstructs an `Env` of byte-window `LazyConstant`s over that same buffer. - This is anon-aligned with the circuit: binder metadata (the `ExprMetaArena`) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-`Defn` hint. The CLI's `name -> addr` map is kept so targets can still be named. - Pure-Lean `getEnv`/`deEnv`/`rsDeEnv` keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (`CheckCmd`) switches to `deEnvAnon`. Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships `rawBytes` windows directly), shard partitioning, decompile, and the compile/commit env builders. Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s, and the lazy run additionally typechecks the target rather than only resolving its address. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

samuelburnham · 2026-06-16T19:28:55Z

ix check --ixe deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load.

Make the Lean Ixon.Env constants lazy and load them without copying:

LazyConstant is now an offset window (buf, off, len) into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized.
New rs_de_env_lazy FFI: Env::parse_lazy_index walks the env once (reusing the existing parser, so every metadata variant incl. CallSite is handled) and returns per-const (addr, offset, len) windows into the buffer Lean passed in, plus name -> addr and per-Defn reducibility hints. No constant body is parsed or copied at load time. deEnvAnon reconstructs an Env of byte-window LazyConstants over that same buffer.
This is anon-aligned with the circuit: binder metadata (the ExprMetaArena) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-Defn hint. The CLI's name -> addr map is kept so targets can still be named.
Pure-Lean getEnv/deEnv/rsDeEnv keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (CheckCmd) switches to deEnvAnon.

Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships rawBytes windows directly), shard partitioning, decompile, and the compile/commit env builders.

Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

Builds on the Rust kernel lazy deserialization from #415

arthurpaulino

Nice!
The "perf(aiur)" tripped me up a bit, but the patch looks good.

Merge the Ix-compiler benchmark and a new Aiur execute/prove benchmark into one workflow (aiur-bench.yml). Per library env it runs 'ix compile' to a .ixe (compile-throughput metrics) then STARK-checks selected constants over that same .ixe via bench-typecheck (execute + prove metrics), combined into one bencher report. bench-typecheck loads the env lazily (Ixon.deEnvAnon, #445) so Mathlib statements check at ~10GB peak RSS instead of the ~100GB eager load. Constants are inline in the workflow matrix. Replaces compile.yml.

`ix check --ixe` deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load. Make the Lean `Ixon.Env` constants lazy and load them without copying: - `LazyConstant` is now an offset window `(buf, off, len)` into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized. - New `rs_de_env_lazy` FFI: `Env::parse_lazy_index` walks the env once (reusing the existing parser, so every metadata variant incl. `CallSite` is handled) and returns per-const `(addr, offset, len)` windows into the buffer Lean passed in, plus `name -> addr` and per-`Defn` reducibility hints. No constant body is parsed or copied at load time. `deEnvAnon` reconstructs an `Env` of byte-window `LazyConstant`s over that same buffer. - This is anon-aligned with the circuit: binder metadata (the `ExprMetaArena`) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-`Defn` hint. The CLI's `name -> addr` map is kept so targets can still be named. - Pure-Lean `getEnv`/`deEnv`/`rsDeEnv` keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (`CheckCmd`) switches to `deEnvAnon`. Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships `rawBytes` windows directly), shard partitioning, decompile, and the compile/commit env builders. Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s, and the lazy run additionally typechecks the target rather than only resolving its address. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

samuelburnham · 2026-06-16T19:28:55Z

ix check --ixe deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load.

Make the Lean Ixon.Env constants lazy and load them without copying:

LazyConstant is now an offset window (buf, off, len) into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized.
New rs_de_env_lazy FFI: Env::parse_lazy_index walks the env once (reusing the existing parser, so every metadata variant incl. CallSite is handled) and returns per-const (addr, offset, len) windows into the buffer Lean passed in, plus name -> addr and per-Defn reducibility hints. No constant body is parsed or copied at load time. deEnvAnon reconstructs an Env of byte-window LazyConstants over that same buffer.
This is anon-aligned with the circuit: binder metadata (the ExprMetaArena) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-Defn hint. The CLI's name -> addr map is kept so targets can still be named.
Pure-Lean getEnv/deEnv/rsDeEnv keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (CheckCmd) switches to deEnvAnon.

Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships rawBytes windows directly), shard partitioning, decompile, and the compile/commit env builders.

Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

Builds on the Rust kernel lazy deserialization from #415

arthurpaulino

Nice!
The "perf(aiur)" tripped me up a bit, but the patch looks good.

Merge the Ix-compiler benchmark and a new Aiur execute/prove benchmark into one workflow (aiur-bench.yml). Per library env it runs 'ix compile' to a .ixe (compile-throughput metrics) then STARK-checks selected constants over that same .ixe via bench-typecheck (execute + prove metrics), combined into one bencher report. bench-typecheck loads the env lazily (Ixon.deEnvAnon, #445) so Mathlib statements check at ~10GB peak RSS instead of the ~100GB eager load. Constants are inline in the workflow matrix. Replaces compile.yml.

`ix check --ixe` deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load. Make the Lean `Ixon.Env` constants lazy and load them without copying: - `LazyConstant` is now an offset window `(buf, off, len)` into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized. - New `rs_de_env_lazy` FFI: `Env::parse_lazy_index` walks the env once (reusing the existing parser, so every metadata variant incl. `CallSite` is handled) and returns per-const `(addr, offset, len)` windows into the buffer Lean passed in, plus `name -> addr` and per-`Defn` reducibility hints. No constant body is parsed or copied at load time. `deEnvAnon` reconstructs an `Env` of byte-window `LazyConstant`s over that same buffer. - This is anon-aligned with the circuit: binder metadata (the `ExprMetaArena`) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-`Defn` hint. The CLI's `name -> addr` map is kept so targets can still be named. - Pure-Lean `getEnv`/`deEnv`/`rsDeEnv` keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (`CheckCmd`) switches to `deEnvAnon`. Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships `rawBytes` windows directly), shard partitioning, decompile, and the compile/commit env builders. Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s, and the lazy run additionally typechecks the target rather than only resolving its address. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

samuelburnham · 2026-06-16T19:28:55Z

ix check --ixe deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load.

Make the Lean Ixon.Env constants lazy and load them without copying:

LazyConstant is now an offset window (buf, off, len) into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized.
New rs_de_env_lazy FFI: Env::parse_lazy_index walks the env once (reusing the existing parser, so every metadata variant incl. CallSite is handled) and returns per-const (addr, offset, len) windows into the buffer Lean passed in, plus name -> addr and per-Defn reducibility hints. No constant body is parsed or copied at load time. deEnvAnon reconstructs an Env of byte-window LazyConstants over that same buffer.
This is anon-aligned with the circuit: binder metadata (the ExprMetaArena) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-Defn hint. The CLI's name -> addr map is kept so targets can still be named.
Pure-Lean getEnv/deEnv/rsDeEnv keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (CheckCmd) switches to deEnvAnon.

Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships rawBytes windows directly), shard partitioning, decompile, and the compile/commit env builders.

Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

Builds on the Rust kernel lazy deserialization from #415

arthurpaulino

Nice!
The "perf(aiur)" tripped me up a bit, but the patch looks good.

Merge the Ix-compiler benchmark and a new Aiur execute/prove benchmark into one workflow (aiur-bench.yml). Per library env it runs 'ix compile' to a .ixe (compile-throughput metrics) then STARK-checks selected constants over that same .ixe via bench-typecheck (execute + prove metrics), combined into one bencher report. bench-typecheck loads the env lazily (Ixon.deEnvAnon, #445) so Mathlib statements check at ~10GB peak RSS instead of the ~100GB eager load. Constants are inline in the workflow matrix. Replaces compile.yml.

`ix check --ixe` deserialized the whole env eagerly: the Rust FFI materialized every constant and the Lean side built an object graph for all of them up front, even to check one lemma. On mathlib.ixe (2.97 GB) that was ~104 GB peak RSS / ~159 s just to load. Make the Lean `Ixon.Env` constants lazy and load them without copying: - `LazyConstant` is now an offset window `(buf, off, len)` into a shared backing buffer (+ optional materialized cache for the build path), parsed on demand. Only the checked closure is ever materialized. - New `rs_de_env_lazy` FFI: `Env::parse_lazy_index` walks the env once (reusing the existing parser, so every metadata variant incl. `CallSite` is handled) and returns per-const `(addr, offset, len)` windows into the buffer Lean passed in, plus `name -> addr` and per-`Defn` reducibility hints. No constant body is parsed or copied at load time. `deEnvAnon` reconstructs an `Env` of byte-window `LazyConstant`s over that same buffer. - This is anon-aligned with the circuit: binder metadata (the `ExprMetaArena`) is parsed-and-discarded, since the typecheck circuit only consumes anonymous constants, blobs, and the per-`Defn` hint. The CLI's `name -> addr` map is kept so targets can still be named. - Pure-Lean `getEnv`/`deEnv`/`rsDeEnv` keep their full-metadata behavior for round-trip and the non-check loaders; only the check path (`CheckCmd`) switches to `deEnvAnon`. Consumers updated for the lazy const type: closure walk, IOBuffer ingress (ships `rawBytes` windows directly), shard partitioning, decompile, and the compile/commit env builders. Measured on mathlib.ixe (same file, same target): 104.5 GB -> 4.5 GB peak RSS, 159 s -> 13.5 s, and the lazy run additionally typechecks the target rather than only resolving its address. Nothing in circuit changes: same closure bytes, same in-circuit blake3 verification, same proof.

samuelburnham force-pushed the sb/aiur-lazy-env branch 2 times, most recently from a845811 to f27fbfbCompare June 16, 2026 19:43

samuelburnham requested a review from arthurpaulino June 16, 2026 19:46

arthurpaulino approved these changes Jun 16, 2026
View reviewed changes

samuelburnham force-pushed the sb/aiur-lazy-env branch from f27fbfb to 061cfe4Compare June 17, 2026 15:45

samuelburnham enabled auto-merge (squash) June 17, 2026 15:45

samuelburnham merged commit e9699ee into mainJun 17, 2026
14 checks passed

samuelburnham deleted the sb/aiur-lazy-env branch June 17, 2026 15:52

samuelburnham changed the title ~~perf(aiur): lazy, zero-copy anon env load for the Aiur check path~~perf(ix): lazy, zero-copy anon env load for the Aiur check pathJun 17, 2026