[NativeAOT] Prototype guarded incremental compilation by awakecoding · Pull Request #132962 · dotnet/runtime

awakecoding · 2026-08-31T06:17:14Z

Summary

Adds a disabled-by-default, internal NativeAOT incremental-compilation prototype for Windows x64 COFF. A single ILC process retains its completed dependency graph, recompiles a prevalidated finite set of changed method bodies, and creates each updated object by patching a copy of the immutable baseline object.

The prototype adds no public API. Incremental rejection is explicit: ILC prints ILC_INCREMENTAL_REJECTED, exits with code 85, removes outputs it created, and requires the caller to start a fresh clean compilation. Ordinary compiler failures are not classified as clean-fallback requests.

Tracking issue: #132977

Implementation

Captures the effective IL provider chain, including generated P/Invoke stubs.
Validates all update assemblies before baseline object emission.
Resets the mutable MethodCodeNode code, GC, EH, debug, local, and dependency state required for recompilation.
Recompiles the union of methods changed by the current and previous updates, so sequential edits and reverts always derive from the original baseline.
Verifies ordered static and conditional dependencies, reasons, marked state, GC info, frame/unwind data, EH state, debug state, symbols, relocations/addends, alignment, COMDAT state, and object locations after code generation.
Records selected COFF fragment locations per compilation and binds the baseline assembly, configuration, and object with SHA-256.
Copies from the same verified baseline handle, validates every non-relocation byte, stages unique same-directory files, flushes them to disk, and publishes without overwrite.
Poisons retained state after any post-mutation failure so a failed attempt cannot be reused.

Supported envelope

Incremental compilation is accepted only for:

A Windows host targeting Windows x64 NativeAOT COFF, single-file compilation, and one primary input.
OptimizationMode.None with exactly one compiler thread.
Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.
Baseline and updated PE files with equal length, MVID, metadata method count, and complete non-body content after masking only timestamp, checksum, debug-directory, strong-name payload, and encoded method-body ranges.
Method bodies with unchanged encoded size, max stack, init-locals, local signature, and EH shape.
Non-constructor, non-generic leaf methods with exactly one marked, non-canonical, non-unboxing, non-foldable MethodCodeNode, no conditional dependencies or EH, and overlayable ECMA IL.
Identical allowed opcode streams where only explicit integer or floating-point constant operands change.
Selected non-COMDAT COFF fragments whose size, alignment, symbols, relocations, addends, GC/frame/EH/debug state, location, bounds, and overlap checks remain valid.

All other cases request a clean compilation.

RDM build-performance impact

Measured on one eligible edit to reachable Program.BuildModuleViewIndex in the RDM RdmNativeAotFast=true workload:

RDM step	Clean path	Retained incremental path	Impact
ILC object generation/update	640.395 s	0.291837 s	2,194.36x faster; 99.9544% less wall time
Native link	36.55 s	36.55 s	unchanged
ILC + native link developer loop	676.945 s (11m 16.9s)	36.842 s	18.37x faster; 94.56% less wall time
Time saved per eligible edit/link iteration	—	640.103 s (10m 40.1s)	derived from the rows above

The 3,744,339,247-byte incremental object exactly matched the clean object with SHA-256 B3140045782498DC4A06F712C2DAA329B732D6340DFCD3D80AD4181E17844206. The update reused 13,455,307 of 13,455,308 object nodes, patched one byte, and allocated 1,568,736 managed bytes.

These are measured component timings. A separate clean RDM publish measured 951.9 s (15m 51.9s), but a comparable complete incremental dotnet publish was not timed. The defensible practical result is the ~36.8-second edit-and-link loop instead of ~10–11 minutes, not a 36.8-second full build.

The first request still pays for the clean compilation and retains roughly 34–36 GiB. The result applies only to edits that pass the narrow safety gate.

Validation

build.cmd clr+libs+host baseline: succeeded with 0 warnings/errors.
build.cmd clr.aot+libs -rc Release -lc Release: succeeded with 0 warnings/errors.
ILCompiler.Compiler.Tests Release: 80 passed, 0 failed, 0 skipped.
Focused IncrementalCompilationTests: 58 passed, 0 failed, 0 skipped.
Dedicated priority-0 Windows-x64 NativeAOT incremental smoke test: runs automatically during BuildNativeAot and succeeded with 0 warnings/errors.
- Incremental edited object and independent clean edited object: 0BE481A1B058F826D4FCD0E013DEFC544BF4382E16CE8C5549EF94AE1F73666F.
- Reverted object and baseline object: 0CB70EB6CAA77ABC4C6F120AE64C1AF3F6370A37AEC4FD54FBF8A96179E44497.
- Explicit exit-85 rejection logging was validated under Windows PowerShell; CFG legs skip the unsupported differential path.
- Build-only comparison objects are written under the intermediate tree and excluded from Helix payloads.

Limitations

Normal compiler-driven source edits commonly change the MVID and therefore request a clean compilation. Productization would require a supported command/API contract, complete cross-process content keys for references/resources/toolchain/JIT/environment, request isolation, eviction and crash recovery, broader invalidation for optimized/inlined/preinitialized/reflection/generic/global facts, discovery of newly dirty nodes, validation or regeneration of linker-affecting side outputs, and compile-time-checked internal seams across the compiler assemblies.

Note

This pull request description was generated with GitHub Copilot.

azure-pipelines · 2026-08-31T06:18:00Z

Azure Pipelines: Successfully started running 4 pipeline(s). 12 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

dotnet-policy-service · 2026-08-31T06:18:41Z

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

jkotas · 2026-08-31T15:08:27Z

What are you trying to achieve with this?

It would be better to start with an issue discussing the experience you would like to see.

jkoritzinsky · 2026-08-31T15:52:03Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

awakecoding · 2026-08-31T16:11:11Z

What are you trying to achieve with this?
It would be better to start with an issue discussing the experience you would like to see.

I am trying to improve incremental build performance for NativeAOT on very large projects like Remote Desktop Manager. It currently takes a good 15 minutes to make a change and rebuild, and memory peaks over 32GB in ILC. I've been told you're supposed to just make managed builds with the warnings for NativeAOT to figure out what to fix for NativeAOT and then just wait on the NativeAOT build, trusting that it works, but this prevents actually trying out the NativeAOT build and repeatedly iterate on it. I know my project (RDM) is huge, but that doesn't mean we can't optimize the build tooling to introduce incremental builds

awakecoding · 2026-08-31T16:33:46Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

Where is this documented?

jkoritzinsky · 2026-08-31T16:50:04Z

You can set the IlcMultiModule property to true to try it out. Once again, no guarantees, this is unsupported, etc.

I also don't know if we actually have the MSBuild targets set up correctly to make this run in an incremental manner so this may need more work for that front.

awakecoding · 2026-08-31T18:21:15Z

@jkotas @jkoritzinsky I have created an issue here for NativeAOT incremental compilation support: #132977

awakecoding · 2026-08-31T20:11:03Z

Thanks — I tried IlcMultiModule on our RDM NativeAOT build. After working around a Windows target-plumbing issue (TargetOS was not forwarded to BuildFrameworkNativeObjects.proj), framework caching did work: 175 framework objects plus Framework.lib were byte/mtime-identical, and an identical publish dropped from 420.7s to 63.1s.

The app targets only compiled the top-level module, not its 509 non-framework references, so the stock link failed with 168 unresolved externals. I also tried custom per-module orchestration: 505/509 modules compiled and all 505 were reused on a warm run, but four modules hit rooting/generic issues and the aggregate link still failed with 1,248 unresolved externals. So this validates framework-level incrementality, but app-module orchestration and multifile generic/reflection support need more work before it is usable for this workload.

Note

This comment was drafted with GitHub Copilot.

MichalStrehovsky · 2026-08-31T22:10:20Z

Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.

This is just the start. An incremental compilation needs to disable all whole program optimizations, including trimming. These optimizations all have butterfly effects that are difficult to capture and reconstruct., A small change in one method might invalidate an optimization done elsewhere.

Once all the applicable optimizations are disabled, an incremental compilation mode gives:

Output that behaves differently from a real optimized native AOT compilation (this applies in the presence of trim/AOT unsafe code - the trimming+optimizations are invisible to trim/AOT safe code)
Subpar quality outputs because all the ways that native AOT could do better through trimming and whole program optimizations are gone.

If the problem is that the RDM codebase has many trimming warnings (I saw Newtonsoft) and instead of fixing the warnings you're rooting assemblies and retesting, you're not going to have a good time with native AOT or trimming and incremental compilation will not help you, unless you intend to ship the unoptimized incremental build.

First step to native AOT conversion is eliminating all the trimming warnings, we document that in the first section. Maybe the doc doesn't explain the gravity of the situation well enough but the "ensure there are no behavior changes by thoroughly testing your app after building as Native AOT" part really means that if there is any trimming warning, you need to retest the entire app after any change due to the butterfly effects. It is not possible to reliably ship a large app that has trimming warnings.

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

awakecoding · 2026-09-01T14:07:53Z

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

I understand where you are going with this, but I respectfully disagree: this means you have to fully port your app for NativeAOT before you can start trying it out despite the warnings. It's important to be able to iterate on the partial port early on to help with prioritization of the work to be done, but also early validation of code paths which differ significantly in a NativeAOT build as opposed to the managed build.

AI made it possible to consider porting RDM to NativeAOT, but it is still a moonshot project. The first blocker we recently resolved was porting all of RDM Windows from DevExpress+WinForms to Avalonia UI, which involved about 1500+ UI components. WinForms is fine for NativeAOT, but DevExpress isn't. That alone took almost a year with AI, and once finally fixed, I could have my first successful launches of the RDM Windows application with NativeAOT despite a ton of build warnings. Not all features work, but it's enough to start iterating and validating by prioritizing what areas of the application should be fixed next for NativeAOT safety.

The second major blocker we have resolved recently is the PowerShell SDK, which is in-process, and will always rely on JIT for obvious reasons. I didn't want to move things out of process because we expose .NET live objects in scripts in-process for some advanced features. I developed my own PowerShell SDK that remains in-process, and offers live object proxies from a NativeAOT-safe .NET application. Here's the custom PowerShell SDK I developed specifically for NativeAOT consumption: https://github.com/Devolutions/multi-pwsh

My point is not that I would like to ship RDM in production with tons of trim-safety warnings remaining. It is that I would like to iterate much faster on experimental (or production) NativeAOT builds. Managed builds differ too much from NativeAOT builds for them to be useful beyond producing warnings. For instance, my managed build of RDM uses the regular PowerShell SDK while the NativeAOT build uses my NativeAOT-safe PowerShell SDK replacement. Several features are also gated for NativeAOT-safety, as I work my way through the huge backlog of things to fix.

But let's say we fast forward to the future at a point where I have fixed all the NativeAOT build warnings. Even then, my NativeAOT builds will still be extremely slow (RDM won't magically become thin and lightweight), so it means I still have to use managed builds for the inner loop. Those managed builds will always significantly differ from a NativeAOT build. There is a point to be made about introducing incremental builds in NativeAOT in places where it can be done.

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds. I want to explore ways we can do it in places where there's a potential gain to be made, and then work on making the non-reusable parts of the build process faster, in hope that the NativeAOT build time becomes manageable. Right now, it's slow even for a CI build, and we need to bring it down.

agocke · 2026-09-01T14:42:46Z

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds.

I don't see how this follows. As Michal mentioned, whole program optimization means that a change in any method can produce changes in completely unrelated methods. For an incremental compilation to be correct this would somehow need to be fully accounted for, and it might end up leading to zero reuse in common cases.

agocke · 2026-09-01T16:45:13Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

awakecoding · 2026-09-01T17:32:18Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

thanks Andy, that explains it much better. I suggest closing the current PR, and keep the discussion going in the high-level NativeAOT incremental build issue: #132977

I would then research how we could take inspiration from rustc internals and come up with a new prototype for incremental builds in .NET NativeAOT does that doesn't disable certain passes. It would obviously be a lot more work, but I'm up for giving it a try.

MichalPetryka · 2026-09-01T17:59:40Z

The only prior art I'm aware of here is rustc.

MSVC does have incremental builds and LTCG, afair they track everything to handle wpo changes.

awakecoding · 2026-08-31T06:17:14Z

Summary

Adds a disabled-by-default, internal NativeAOT incremental-compilation prototype for Windows x64 COFF. A single ILC process retains its completed dependency graph, recompiles a prevalidated finite set of changed method bodies, and creates each updated object by patching a copy of the immutable baseline object.

The prototype adds no public API. Incremental rejection is explicit: ILC prints ILC_INCREMENTAL_REJECTED, exits with code 85, removes outputs it created, and requires the caller to start a fresh clean compilation. Ordinary compiler failures are not classified as clean-fallback requests.

Tracking issue: #132977

Implementation

Captures the effective IL provider chain, including generated P/Invoke stubs.
Validates all update assemblies before baseline object emission.
Resets the mutable MethodCodeNode code, GC, EH, debug, local, and dependency state required for recompilation.
Recompiles the union of methods changed by the current and previous updates, so sequential edits and reverts always derive from the original baseline.
Verifies ordered static and conditional dependencies, reasons, marked state, GC info, frame/unwind data, EH state, debug state, symbols, relocations/addends, alignment, COMDAT state, and object locations after code generation.
Records selected COFF fragment locations per compilation and binds the baseline assembly, configuration, and object with SHA-256.
Copies from the same verified baseline handle, validates every non-relocation byte, stages unique same-directory files, flushes them to disk, and publishes without overwrite.
Poisons retained state after any post-mutation failure so a failed attempt cannot be reused.

Supported envelope

Incremental compilation is accepted only for:

A Windows host targeting Windows x64 NativeAOT COFF, single-file compilation, and one primary input.
OptimizationMode.None with exactly one compiler thread.
Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.
Baseline and updated PE files with equal length, MVID, metadata method count, and complete non-body content after masking only timestamp, checksum, debug-directory, strong-name payload, and encoded method-body ranges.
Method bodies with unchanged encoded size, max stack, init-locals, local signature, and EH shape.
Non-constructor, non-generic leaf methods with exactly one marked, non-canonical, non-unboxing, non-foldable MethodCodeNode, no conditional dependencies or EH, and overlayable ECMA IL.
Identical allowed opcode streams where only explicit integer or floating-point constant operands change.
Selected non-COMDAT COFF fragments whose size, alignment, symbols, relocations, addends, GC/frame/EH/debug state, location, bounds, and overlap checks remain valid.

All other cases request a clean compilation.

RDM build-performance impact

Measured on one eligible edit to reachable Program.BuildModuleViewIndex in the RDM RdmNativeAotFast=true workload:

RDM step	Clean path	Retained incremental path	Impact
ILC object generation/update	640.395 s	0.291837 s	2,194.36x faster; 99.9544% less wall time
Native link	36.55 s	36.55 s	unchanged
ILC + native link developer loop	676.945 s (11m 16.9s)	36.842 s	18.37x faster; 94.56% less wall time
Time saved per eligible edit/link iteration	—	640.103 s (10m 40.1s)	derived from the rows above

The 3,744,339,247-byte incremental object exactly matched the clean object with SHA-256 B3140045782498DC4A06F712C2DAA329B732D6340DFCD3D80AD4181E17844206. The update reused 13,455,307 of 13,455,308 object nodes, patched one byte, and allocated 1,568,736 managed bytes.

These are measured component timings. A separate clean RDM publish measured 951.9 s (15m 51.9s), but a comparable complete incremental dotnet publish was not timed. The defensible practical result is the ~36.8-second edit-and-link loop instead of ~10–11 minutes, not a 36.8-second full build.

The first request still pays for the clean compilation and retains roughly 34–36 GiB. The result applies only to edits that pass the narrow safety gate.

Validation

build.cmd clr+libs+host baseline: succeeded with 0 warnings/errors.
build.cmd clr.aot+libs -rc Release -lc Release: succeeded with 0 warnings/errors.
ILCompiler.Compiler.Tests Release: 80 passed, 0 failed, 0 skipped.
Focused IncrementalCompilationTests: 58 passed, 0 failed, 0 skipped.
Dedicated priority-0 Windows-x64 NativeAOT incremental smoke test: runs automatically during BuildNativeAot and succeeded with 0 warnings/errors.
- Incremental edited object and independent clean edited object: 0BE481A1B058F826D4FCD0E013DEFC544BF4382E16CE8C5549EF94AE1F73666F.
- Reverted object and baseline object: 0CB70EB6CAA77ABC4C6F120AE64C1AF3F6370A37AEC4FD54FBF8A96179E44497.
- Explicit exit-85 rejection logging was validated under Windows PowerShell; CFG legs skip the unsupported differential path.
- Build-only comparison objects are written under the intermediate tree and excluded from Helix payloads.

Limitations

Normal compiler-driven source edits commonly change the MVID and therefore request a clean compilation. Productization would require a supported command/API contract, complete cross-process content keys for references/resources/toolchain/JIT/environment, request isolation, eviction and crash recovery, broader invalidation for optimized/inlined/preinitialized/reflection/generic/global facts, discovery of newly dirty nodes, validation or regeneration of linker-affecting side outputs, and compile-time-checked internal seams across the compiler assemblies.

Note

This pull request description was generated with GitHub Copilot.

azure-pipelines · 2026-08-31T06:18:00Z

Azure Pipelines: Successfully started running 4 pipeline(s). 12 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

dotnet-policy-service · 2026-08-31T06:18:41Z

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

jkotas · 2026-08-31T15:08:27Z

What are you trying to achieve with this?

It would be better to start with an issue discussing the experience you would like to see.

jkoritzinsky · 2026-08-31T15:52:03Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

awakecoding · 2026-08-31T16:11:11Z

What are you trying to achieve with this?
It would be better to start with an issue discussing the experience you would like to see.

I am trying to improve incremental build performance for NativeAOT on very large projects like Remote Desktop Manager. It currently takes a good 15 minutes to make a change and rebuild, and memory peaks over 32GB in ILC. I've been told you're supposed to just make managed builds with the warnings for NativeAOT to figure out what to fix for NativeAOT and then just wait on the NativeAOT build, trusting that it works, but this prevents actually trying out the NativeAOT build and repeatedly iterate on it. I know my project (RDM) is huge, but that doesn't mean we can't optimize the build tooling to introduce incremental builds

awakecoding · 2026-08-31T16:33:46Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

Where is this documented?

jkoritzinsky · 2026-08-31T16:50:04Z

You can set the IlcMultiModule property to true to try it out. Once again, no guarantees, this is unsupported, etc.

I also don't know if we actually have the MSBuild targets set up correctly to make this run in an incremental manner so this may need more work for that front.

awakecoding · 2026-08-31T18:21:15Z

@jkotas @jkoritzinsky I have created an issue here for NativeAOT incremental compilation support: #132977

awakecoding · 2026-08-31T20:11:03Z

Thanks — I tried IlcMultiModule on our RDM NativeAOT build. After working around a Windows target-plumbing issue (TargetOS was not forwarded to BuildFrameworkNativeObjects.proj), framework caching did work: 175 framework objects plus Framework.lib were byte/mtime-identical, and an identical publish dropped from 420.7s to 63.1s.

The app targets only compiled the top-level module, not its 509 non-framework references, so the stock link failed with 168 unresolved externals. I also tried custom per-module orchestration: 505/509 modules compiled and all 505 were reused on a warm run, but four modules hit rooting/generic issues and the aggregate link still failed with 1,248 unresolved externals. So this validates framework-level incrementality, but app-module orchestration and multifile generic/reflection support need more work before it is usable for this workload.

Note

This comment was drafted with GitHub Copilot.

MichalStrehovsky · 2026-08-31T22:10:20Z

Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.

This is just the start. An incremental compilation needs to disable all whole program optimizations, including trimming. These optimizations all have butterfly effects that are difficult to capture and reconstruct., A small change in one method might invalidate an optimization done elsewhere.

Once all the applicable optimizations are disabled, an incremental compilation mode gives:

Output that behaves differently from a real optimized native AOT compilation (this applies in the presence of trim/AOT unsafe code - the trimming+optimizations are invisible to trim/AOT safe code)
Subpar quality outputs because all the ways that native AOT could do better through trimming and whole program optimizations are gone.

If the problem is that the RDM codebase has many trimming warnings (I saw Newtonsoft) and instead of fixing the warnings you're rooting assemblies and retesting, you're not going to have a good time with native AOT or trimming and incremental compilation will not help you, unless you intend to ship the unoptimized incremental build.

First step to native AOT conversion is eliminating all the trimming warnings, we document that in the first section. Maybe the doc doesn't explain the gravity of the situation well enough but the "ensure there are no behavior changes by thoroughly testing your app after building as Native AOT" part really means that if there is any trimming warning, you need to retest the entire app after any change due to the butterfly effects. It is not possible to reliably ship a large app that has trimming warnings.

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

awakecoding · 2026-09-01T14:07:53Z

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

I understand where you are going with this, but I respectfully disagree: this means you have to fully port your app for NativeAOT before you can start trying it out despite the warnings. It's important to be able to iterate on the partial port early on to help with prioritization of the work to be done, but also early validation of code paths which differ significantly in a NativeAOT build as opposed to the managed build.

AI made it possible to consider porting RDM to NativeAOT, but it is still a moonshot project. The first blocker we recently resolved was porting all of RDM Windows from DevExpress+WinForms to Avalonia UI, which involved about 1500+ UI components. WinForms is fine for NativeAOT, but DevExpress isn't. That alone took almost a year with AI, and once finally fixed, I could have my first successful launches of the RDM Windows application with NativeAOT despite a ton of build warnings. Not all features work, but it's enough to start iterating and validating by prioritizing what areas of the application should be fixed next for NativeAOT safety.

The second major blocker we have resolved recently is the PowerShell SDK, which is in-process, and will always rely on JIT for obvious reasons. I didn't want to move things out of process because we expose .NET live objects in scripts in-process for some advanced features. I developed my own PowerShell SDK that remains in-process, and offers live object proxies from a NativeAOT-safe .NET application. Here's the custom PowerShell SDK I developed specifically for NativeAOT consumption: https://github.com/Devolutions/multi-pwsh

My point is not that I would like to ship RDM in production with tons of trim-safety warnings remaining. It is that I would like to iterate much faster on experimental (or production) NativeAOT builds. Managed builds differ too much from NativeAOT builds for them to be useful beyond producing warnings. For instance, my managed build of RDM uses the regular PowerShell SDK while the NativeAOT build uses my NativeAOT-safe PowerShell SDK replacement. Several features are also gated for NativeAOT-safety, as I work my way through the huge backlog of things to fix.

But let's say we fast forward to the future at a point where I have fixed all the NativeAOT build warnings. Even then, my NativeAOT builds will still be extremely slow (RDM won't magically become thin and lightweight), so it means I still have to use managed builds for the inner loop. Those managed builds will always significantly differ from a NativeAOT build. There is a point to be made about introducing incremental builds in NativeAOT in places where it can be done.

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds. I want to explore ways we can do it in places where there's a potential gain to be made, and then work on making the non-reusable parts of the build process faster, in hope that the NativeAOT build time becomes manageable. Right now, it's slow even for a CI build, and we need to bring it down.

agocke · 2026-09-01T14:42:46Z

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds.

I don't see how this follows. As Michal mentioned, whole program optimization means that a change in any method can produce changes in completely unrelated methods. For an incremental compilation to be correct this would somehow need to be fully accounted for, and it might end up leading to zero reuse in common cases.

agocke · 2026-09-01T16:45:13Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

awakecoding · 2026-09-01T17:32:18Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

thanks Andy, that explains it much better. I suggest closing the current PR, and keep the discussion going in the high-level NativeAOT incremental build issue: #132977

I would then research how we could take inspiration from rustc internals and come up with a new prototype for incremental builds in .NET NativeAOT does that doesn't disable certain passes. It would obviously be a lot more work, but I'm up for giving it a try.

MichalPetryka · 2026-09-01T17:59:40Z

The only prior art I'm aware of here is rustc.

MSVC does have incremental builds and LTCG, afair they track everything to handle wpo changes.

awakecoding · 2026-08-31T06:17:14Z

Summary

Adds a disabled-by-default, internal NativeAOT incremental-compilation prototype for Windows x64 COFF. A single ILC process retains its completed dependency graph, recompiles a prevalidated finite set of changed method bodies, and creates each updated object by patching a copy of the immutable baseline object.

The prototype adds no public API. Incremental rejection is explicit: ILC prints ILC_INCREMENTAL_REJECTED, exits with code 85, removes outputs it created, and requires the caller to start a fresh clean compilation. Ordinary compiler failures are not classified as clean-fallback requests.

Tracking issue: #132977

Implementation

Captures the effective IL provider chain, including generated P/Invoke stubs.
Validates all update assemblies before baseline object emission.
Resets the mutable MethodCodeNode code, GC, EH, debug, local, and dependency state required for recompilation.
Recompiles the union of methods changed by the current and previous updates, so sequential edits and reverts always derive from the original baseline.
Verifies ordered static and conditional dependencies, reasons, marked state, GC info, frame/unwind data, EH state, debug state, symbols, relocations/addends, alignment, COMDAT state, and object locations after code generation.
Records selected COFF fragment locations per compilation and binds the baseline assembly, configuration, and object with SHA-256.
Copies from the same verified baseline handle, validates every non-relocation byte, stages unique same-directory files, flushes them to disk, and publishes without overwrite.
Poisons retained state after any post-mutation failure so a failed attempt cannot be reused.

Supported envelope

Incremental compilation is accepted only for:

A Windows host targeting Windows x64 NativeAOT COFF, single-file compilation, and one primary input.
OptimizationMode.None with exactly one compiler thread.
Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.
Baseline and updated PE files with equal length, MVID, metadata method count, and complete non-body content after masking only timestamp, checksum, debug-directory, strong-name payload, and encoded method-body ranges.
Method bodies with unchanged encoded size, max stack, init-locals, local signature, and EH shape.
Non-constructor, non-generic leaf methods with exactly one marked, non-canonical, non-unboxing, non-foldable MethodCodeNode, no conditional dependencies or EH, and overlayable ECMA IL.
Identical allowed opcode streams where only explicit integer or floating-point constant operands change.
Selected non-COMDAT COFF fragments whose size, alignment, symbols, relocations, addends, GC/frame/EH/debug state, location, bounds, and overlap checks remain valid.

All other cases request a clean compilation.

RDM build-performance impact

Measured on one eligible edit to reachable Program.BuildModuleViewIndex in the RDM RdmNativeAotFast=true workload:

RDM step	Clean path	Retained incremental path	Impact
ILC object generation/update	640.395 s	0.291837 s	2,194.36x faster; 99.9544% less wall time
Native link	36.55 s	36.55 s	unchanged
ILC + native link developer loop	676.945 s (11m 16.9s)	36.842 s	18.37x faster; 94.56% less wall time
Time saved per eligible edit/link iteration	—	640.103 s (10m 40.1s)	derived from the rows above

The 3,744,339,247-byte incremental object exactly matched the clean object with SHA-256 B3140045782498DC4A06F712C2DAA329B732D6340DFCD3D80AD4181E17844206. The update reused 13,455,307 of 13,455,308 object nodes, patched one byte, and allocated 1,568,736 managed bytes.

These are measured component timings. A separate clean RDM publish measured 951.9 s (15m 51.9s), but a comparable complete incremental dotnet publish was not timed. The defensible practical result is the ~36.8-second edit-and-link loop instead of ~10–11 minutes, not a 36.8-second full build.

The first request still pays for the clean compilation and retains roughly 34–36 GiB. The result applies only to edits that pass the narrow safety gate.

Validation

build.cmd clr+libs+host baseline: succeeded with 0 warnings/errors.
build.cmd clr.aot+libs -rc Release -lc Release: succeeded with 0 warnings/errors.
ILCompiler.Compiler.Tests Release: 80 passed, 0 failed, 0 skipped.
Focused IncrementalCompilationTests: 58 passed, 0 failed, 0 skipped.
Dedicated priority-0 Windows-x64 NativeAOT incremental smoke test: runs automatically during BuildNativeAot and succeeded with 0 warnings/errors.
- Incremental edited object and independent clean edited object: 0BE481A1B058F826D4FCD0E013DEFC544BF4382E16CE8C5549EF94AE1F73666F.
- Reverted object and baseline object: 0CB70EB6CAA77ABC4C6F120AE64C1AF3F6370A37AEC4FD54FBF8A96179E44497.
- Explicit exit-85 rejection logging was validated under Windows PowerShell; CFG legs skip the unsupported differential path.
- Build-only comparison objects are written under the intermediate tree and excluded from Helix payloads.

Limitations

Normal compiler-driven source edits commonly change the MVID and therefore request a clean compilation. Productization would require a supported command/API contract, complete cross-process content keys for references/resources/toolchain/JIT/environment, request isolation, eviction and crash recovery, broader invalidation for optimized/inlined/preinitialized/reflection/generic/global facts, discovery of newly dirty nodes, validation or regeneration of linker-affecting side outputs, and compile-time-checked internal seams across the compiler assemblies.

Note

This pull request description was generated with GitHub Copilot.

azure-pipelines · 2026-08-31T06:18:00Z

Azure Pipelines: Successfully started running 4 pipeline(s). 12 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

dotnet-policy-service · 2026-08-31T06:18:41Z

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

jkotas · 2026-08-31T15:08:27Z

What are you trying to achieve with this?

It would be better to start with an issue discussing the experience you would like to see.

jkoritzinsky · 2026-08-31T15:52:03Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

awakecoding · 2026-08-31T16:11:11Z

What are you trying to achieve with this?
It would be better to start with an issue discussing the experience you would like to see.

I am trying to improve incremental build performance for NativeAOT on very large projects like Remote Desktop Manager. It currently takes a good 15 minutes to make a change and rebuild, and memory peaks over 32GB in ILC. I've been told you're supposed to just make managed builds with the warnings for NativeAOT to figure out what to fix for NativeAOT and then just wait on the NativeAOT build, trusting that it works, but this prevents actually trying out the NativeAOT build and repeatedly iterate on it. I know my project (RDM) is huge, but that doesn't mean we can't optimize the build tooling to introduce incremental builds

awakecoding · 2026-08-31T16:33:46Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

Where is this documented?

jkoritzinsky · 2026-08-31T16:50:04Z

You can set the IlcMultiModule property to true to try it out. Once again, no guarantees, this is unsupported, etc.

I also don't know if we actually have the MSBuild targets set up correctly to make this run in an incremental manner so this may need more work for that front.

awakecoding · 2026-08-31T18:21:15Z

@jkotas @jkoritzinsky I have created an issue here for NativeAOT incremental compilation support: #132977

awakecoding · 2026-08-31T20:11:03Z

Thanks — I tried IlcMultiModule on our RDM NativeAOT build. After working around a Windows target-plumbing issue (TargetOS was not forwarded to BuildFrameworkNativeObjects.proj), framework caching did work: 175 framework objects plus Framework.lib were byte/mtime-identical, and an identical publish dropped from 420.7s to 63.1s.

The app targets only compiled the top-level module, not its 509 non-framework references, so the stock link failed with 168 unresolved externals. I also tried custom per-module orchestration: 505/509 modules compiled and all 505 were reused on a warm run, but four modules hit rooting/generic issues and the aggregate link still failed with 1,248 unresolved externals. So this validates framework-level incrementality, but app-module orchestration and multifile generic/reflection support need more work before it is usable for this workload.

Note

This comment was drafted with GitHub Copilot.

MichalStrehovsky · 2026-08-31T22:10:20Z

Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.

This is just the start. An incremental compilation needs to disable all whole program optimizations, including trimming. These optimizations all have butterfly effects that are difficult to capture and reconstruct., A small change in one method might invalidate an optimization done elsewhere.

Once all the applicable optimizations are disabled, an incremental compilation mode gives:

Output that behaves differently from a real optimized native AOT compilation (this applies in the presence of trim/AOT unsafe code - the trimming+optimizations are invisible to trim/AOT safe code)
Subpar quality outputs because all the ways that native AOT could do better through trimming and whole program optimizations are gone.

If the problem is that the RDM codebase has many trimming warnings (I saw Newtonsoft) and instead of fixing the warnings you're rooting assemblies and retesting, you're not going to have a good time with native AOT or trimming and incremental compilation will not help you, unless you intend to ship the unoptimized incremental build.

First step to native AOT conversion is eliminating all the trimming warnings, we document that in the first section. Maybe the doc doesn't explain the gravity of the situation well enough but the "ensure there are no behavior changes by thoroughly testing your app after building as Native AOT" part really means that if there is any trimming warning, you need to retest the entire app after any change due to the butterfly effects. It is not possible to reliably ship a large app that has trimming warnings.

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

awakecoding · 2026-09-01T14:07:53Z

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

I understand where you are going with this, but I respectfully disagree: this means you have to fully port your app for NativeAOT before you can start trying it out despite the warnings. It's important to be able to iterate on the partial port early on to help with prioritization of the work to be done, but also early validation of code paths which differ significantly in a NativeAOT build as opposed to the managed build.

AI made it possible to consider porting RDM to NativeAOT, but it is still a moonshot project. The first blocker we recently resolved was porting all of RDM Windows from DevExpress+WinForms to Avalonia UI, which involved about 1500+ UI components. WinForms is fine for NativeAOT, but DevExpress isn't. That alone took almost a year with AI, and once finally fixed, I could have my first successful launches of the RDM Windows application with NativeAOT despite a ton of build warnings. Not all features work, but it's enough to start iterating and validating by prioritizing what areas of the application should be fixed next for NativeAOT safety.

The second major blocker we have resolved recently is the PowerShell SDK, which is in-process, and will always rely on JIT for obvious reasons. I didn't want to move things out of process because we expose .NET live objects in scripts in-process for some advanced features. I developed my own PowerShell SDK that remains in-process, and offers live object proxies from a NativeAOT-safe .NET application. Here's the custom PowerShell SDK I developed specifically for NativeAOT consumption: https://github.com/Devolutions/multi-pwsh

My point is not that I would like to ship RDM in production with tons of trim-safety warnings remaining. It is that I would like to iterate much faster on experimental (or production) NativeAOT builds. Managed builds differ too much from NativeAOT builds for them to be useful beyond producing warnings. For instance, my managed build of RDM uses the regular PowerShell SDK while the NativeAOT build uses my NativeAOT-safe PowerShell SDK replacement. Several features are also gated for NativeAOT-safety, as I work my way through the huge backlog of things to fix.

But let's say we fast forward to the future at a point where I have fixed all the NativeAOT build warnings. Even then, my NativeAOT builds will still be extremely slow (RDM won't magically become thin and lightweight), so it means I still have to use managed builds for the inner loop. Those managed builds will always significantly differ from a NativeAOT build. There is a point to be made about introducing incremental builds in NativeAOT in places where it can be done.

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds. I want to explore ways we can do it in places where there's a potential gain to be made, and then work on making the non-reusable parts of the build process faster, in hope that the NativeAOT build time becomes manageable. Right now, it's slow even for a CI build, and we need to bring it down.

agocke · 2026-09-01T14:42:46Z

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds.

I don't see how this follows. As Michal mentioned, whole program optimization means that a change in any method can produce changes in completely unrelated methods. For an incremental compilation to be correct this would somehow need to be fully accounted for, and it might end up leading to zero reuse in common cases.

agocke · 2026-09-01T16:45:13Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

awakecoding · 2026-09-01T17:32:18Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

thanks Andy, that explains it much better. I suggest closing the current PR, and keep the discussion going in the high-level NativeAOT incremental build issue: #132977

I would then research how we could take inspiration from rustc internals and come up with a new prototype for incremental builds in .NET NativeAOT does that doesn't disable certain passes. It would obviously be a lot more work, but I'm up for giving it a try.

MichalPetryka · 2026-09-01T17:59:40Z

The only prior art I'm aware of here is rustc.

MSVC does have incremental builds and LTCG, afair they track everything to handle wpo changes.

awakecoding · 2026-08-31T06:17:14Z

Summary

Adds a disabled-by-default, internal NativeAOT incremental-compilation prototype for Windows x64 COFF. A single ILC process retains its completed dependency graph, recompiles a prevalidated finite set of changed method bodies, and creates each updated object by patching a copy of the immutable baseline object.

The prototype adds no public API. Incremental rejection is explicit: ILC prints ILC_INCREMENTAL_REJECTED, exits with code 85, removes outputs it created, and requires the caller to start a fresh clean compilation. Ordinary compiler failures are not classified as clean-fallback requests.

Tracking issue: #132977

Implementation

Captures the effective IL provider chain, including generated P/Invoke stubs.
Validates all update assemblies before baseline object emission.
Resets the mutable MethodCodeNode code, GC, EH, debug, local, and dependency state required for recompilation.
Recompiles the union of methods changed by the current and previous updates, so sequential edits and reverts always derive from the original baseline.
Verifies ordered static and conditional dependencies, reasons, marked state, GC info, frame/unwind data, EH state, debug state, symbols, relocations/addends, alignment, COMDAT state, and object locations after code generation.
Records selected COFF fragment locations per compilation and binds the baseline assembly, configuration, and object with SHA-256.
Copies from the same verified baseline handle, validates every non-relocation byte, stages unique same-directory files, flushes them to disk, and publishes without overwrite.
Poisons retained state after any post-mutation failure so a failed attempt cannot be reused.

Supported envelope

Incremental compilation is accepted only for:

A Windows host targeting Windows x64 NativeAOT COFF, single-file compilation, and one primary input.
OptimizationMode.None with exactly one compiler thread.
Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.
Baseline and updated PE files with equal length, MVID, metadata method count, and complete non-body content after masking only timestamp, checksum, debug-directory, strong-name payload, and encoded method-body ranges.
Method bodies with unchanged encoded size, max stack, init-locals, local signature, and EH shape.
Non-constructor, non-generic leaf methods with exactly one marked, non-canonical, non-unboxing, non-foldable MethodCodeNode, no conditional dependencies or EH, and overlayable ECMA IL.
Identical allowed opcode streams where only explicit integer or floating-point constant operands change.
Selected non-COMDAT COFF fragments whose size, alignment, symbols, relocations, addends, GC/frame/EH/debug state, location, bounds, and overlap checks remain valid.

All other cases request a clean compilation.

RDM build-performance impact

Measured on one eligible edit to reachable Program.BuildModuleViewIndex in the RDM RdmNativeAotFast=true workload:

RDM step	Clean path	Retained incremental path	Impact
ILC object generation/update	640.395 s	0.291837 s	2,194.36x faster; 99.9544% less wall time
Native link	36.55 s	36.55 s	unchanged
ILC + native link developer loop	676.945 s (11m 16.9s)	36.842 s	18.37x faster; 94.56% less wall time
Time saved per eligible edit/link iteration	—	640.103 s (10m 40.1s)	derived from the rows above

The 3,744,339,247-byte incremental object exactly matched the clean object with SHA-256 B3140045782498DC4A06F712C2DAA329B732D6340DFCD3D80AD4181E17844206. The update reused 13,455,307 of 13,455,308 object nodes, patched one byte, and allocated 1,568,736 managed bytes.

These are measured component timings. A separate clean RDM publish measured 951.9 s (15m 51.9s), but a comparable complete incremental dotnet publish was not timed. The defensible practical result is the ~36.8-second edit-and-link loop instead of ~10–11 minutes, not a 36.8-second full build.

The first request still pays for the clean compilation and retains roughly 34–36 GiB. The result applies only to edits that pass the narrow safety gate.

Validation

build.cmd clr+libs+host baseline: succeeded with 0 warnings/errors.
build.cmd clr.aot+libs -rc Release -lc Release: succeeded with 0 warnings/errors.
ILCompiler.Compiler.Tests Release: 80 passed, 0 failed, 0 skipped.
Focused IncrementalCompilationTests: 58 passed, 0 failed, 0 skipped.
Dedicated priority-0 Windows-x64 NativeAOT incremental smoke test: runs automatically during BuildNativeAot and succeeded with 0 warnings/errors.
- Incremental edited object and independent clean edited object: 0BE481A1B058F826D4FCD0E013DEFC544BF4382E16CE8C5549EF94AE1F73666F.
- Reverted object and baseline object: 0CB70EB6CAA77ABC4C6F120AE64C1AF3F6370A37AEC4FD54FBF8A96179E44497.
- Explicit exit-85 rejection logging was validated under Windows PowerShell; CFG legs skip the unsupported differential path.
- Build-only comparison objects are written under the intermediate tree and excluded from Helix payloads.

Limitations

Normal compiler-driven source edits commonly change the MVID and therefore request a clean compilation. Productization would require a supported command/API contract, complete cross-process content keys for references/resources/toolchain/JIT/environment, request isolation, eviction and crash recovery, broader invalidation for optimized/inlined/preinitialized/reflection/generic/global facts, discovery of newly dirty nodes, validation or regeneration of linker-affecting side outputs, and compile-time-checked internal seams across the compiler assemblies.

Note

This pull request description was generated with GitHub Copilot.

azure-pipelines · 2026-08-31T06:18:00Z

Azure Pipelines: Successfully started running 4 pipeline(s). 12 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

dotnet-policy-service · 2026-08-31T06:18:41Z

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

jkotas · 2026-08-31T15:08:27Z

What are you trying to achieve with this?

It would be better to start with an issue discussing the experience you would like to see.

jkoritzinsky · 2026-08-31T15:52:03Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

awakecoding · 2026-08-31T16:11:11Z

What are you trying to achieve with this?
It would be better to start with an issue discussing the experience you would like to see.

I am trying to improve incremental build performance for NativeAOT on very large projects like Remote Desktop Manager. It currently takes a good 15 minutes to make a change and rebuild, and memory peaks over 32GB in ILC. I've been told you're supposed to just make managed builds with the warnings for NativeAOT to figure out what to fix for NativeAOT and then just wait on the NativeAOT build, trusting that it works, but this prevents actually trying out the NativeAOT build and repeatedly iterate on it. I know my project (RDM) is huge, but that doesn't mean we can't optimize the build tooling to introduce incremental builds

awakecoding · 2026-08-31T16:33:46Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

Where is this documented?

jkoritzinsky · 2026-08-31T16:50:04Z

You can set the IlcMultiModule property to true to try it out. Once again, no guarantees, this is unsupported, etc.

I also don't know if we actually have the MSBuild targets set up correctly to make this run in an incremental manner so this may need more work for that front.

awakecoding · 2026-08-31T18:21:15Z

@jkotas @jkoritzinsky I have created an issue here for NativeAOT incremental compilation support: #132977

awakecoding · 2026-08-31T20:11:03Z

Thanks — I tried IlcMultiModule on our RDM NativeAOT build. After working around a Windows target-plumbing issue (TargetOS was not forwarded to BuildFrameworkNativeObjects.proj), framework caching did work: 175 framework objects plus Framework.lib were byte/mtime-identical, and an identical publish dropped from 420.7s to 63.1s.

The app targets only compiled the top-level module, not its 509 non-framework references, so the stock link failed with 168 unresolved externals. I also tried custom per-module orchestration: 505/509 modules compiled and all 505 were reused on a warm run, but four modules hit rooting/generic issues and the aggregate link still failed with 1,248 unresolved externals. So this validates framework-level incrementality, but app-module orchestration and multifile generic/reflection support need more work before it is usable for this workload.

Note

This comment was drafted with GitHub Copilot.

MichalStrehovsky · 2026-08-31T22:10:20Z

Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.

This is just the start. An incremental compilation needs to disable all whole program optimizations, including trimming. These optimizations all have butterfly effects that are difficult to capture and reconstruct., A small change in one method might invalidate an optimization done elsewhere.

Once all the applicable optimizations are disabled, an incremental compilation mode gives:

Output that behaves differently from a real optimized native AOT compilation (this applies in the presence of trim/AOT unsafe code - the trimming+optimizations are invisible to trim/AOT safe code)
Subpar quality outputs because all the ways that native AOT could do better through trimming and whole program optimizations are gone.

If the problem is that the RDM codebase has many trimming warnings (I saw Newtonsoft) and instead of fixing the warnings you're rooting assemblies and retesting, you're not going to have a good time with native AOT or trimming and incremental compilation will not help you, unless you intend to ship the unoptimized incremental build.

First step to native AOT conversion is eliminating all the trimming warnings, we document that in the first section. Maybe the doc doesn't explain the gravity of the situation well enough but the "ensure there are no behavior changes by thoroughly testing your app after building as Native AOT" part really means that if there is any trimming warning, you need to retest the entire app after any change due to the butterfly effects. It is not possible to reliably ship a large app that has trimming warnings.

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

awakecoding · 2026-09-01T14:07:53Z

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

I understand where you are going with this, but I respectfully disagree: this means you have to fully port your app for NativeAOT before you can start trying it out despite the warnings. It's important to be able to iterate on the partial port early on to help with prioritization of the work to be done, but also early validation of code paths which differ significantly in a NativeAOT build as opposed to the managed build.

AI made it possible to consider porting RDM to NativeAOT, but it is still a moonshot project. The first blocker we recently resolved was porting all of RDM Windows from DevExpress+WinForms to Avalonia UI, which involved about 1500+ UI components. WinForms is fine for NativeAOT, but DevExpress isn't. That alone took almost a year with AI, and once finally fixed, I could have my first successful launches of the RDM Windows application with NativeAOT despite a ton of build warnings. Not all features work, but it's enough to start iterating and validating by prioritizing what areas of the application should be fixed next for NativeAOT safety.

The second major blocker we have resolved recently is the PowerShell SDK, which is in-process, and will always rely on JIT for obvious reasons. I didn't want to move things out of process because we expose .NET live objects in scripts in-process for some advanced features. I developed my own PowerShell SDK that remains in-process, and offers live object proxies from a NativeAOT-safe .NET application. Here's the custom PowerShell SDK I developed specifically for NativeAOT consumption: https://github.com/Devolutions/multi-pwsh

My point is not that I would like to ship RDM in production with tons of trim-safety warnings remaining. It is that I would like to iterate much faster on experimental (or production) NativeAOT builds. Managed builds differ too much from NativeAOT builds for them to be useful beyond producing warnings. For instance, my managed build of RDM uses the regular PowerShell SDK while the NativeAOT build uses my NativeAOT-safe PowerShell SDK replacement. Several features are also gated for NativeAOT-safety, as I work my way through the huge backlog of things to fix.

But let's say we fast forward to the future at a point where I have fixed all the NativeAOT build warnings. Even then, my NativeAOT builds will still be extremely slow (RDM won't magically become thin and lightweight), so it means I still have to use managed builds for the inner loop. Those managed builds will always significantly differ from a NativeAOT build. There is a point to be made about introducing incremental builds in NativeAOT in places where it can be done.

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds. I want to explore ways we can do it in places where there's a potential gain to be made, and then work on making the non-reusable parts of the build process faster, in hope that the NativeAOT build time becomes manageable. Right now, it's slow even for a CI build, and we need to bring it down.

agocke · 2026-09-01T14:42:46Z

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds.

I don't see how this follows. As Michal mentioned, whole program optimization means that a change in any method can produce changes in completely unrelated methods. For an incremental compilation to be correct this would somehow need to be fully accounted for, and it might end up leading to zero reuse in common cases.

agocke · 2026-09-01T16:45:13Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

awakecoding · 2026-09-01T17:32:18Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

thanks Andy, that explains it much better. I suggest closing the current PR, and keep the discussion going in the high-level NativeAOT incremental build issue: #132977

I would then research how we could take inspiration from rustc internals and come up with a new prototype for incremental builds in .NET NativeAOT does that doesn't disable certain passes. It would obviously be a lot more work, but I'm up for giving it a try.

MichalPetryka · 2026-09-01T17:59:40Z

The only prior art I'm aware of here is rustc.

MSVC does have incremental builds and LTCG, afair they track everything to handle wpo changes.

awakecoding · 2026-08-31T06:17:14Z

Summary

Adds a disabled-by-default, internal NativeAOT incremental-compilation prototype for Windows x64 COFF. A single ILC process retains its completed dependency graph, recompiles a prevalidated finite set of changed method bodies, and creates each updated object by patching a copy of the immutable baseline object.

The prototype adds no public API. Incremental rejection is explicit: ILC prints ILC_INCREMENTAL_REJECTED, exits with code 85, removes outputs it created, and requires the caller to start a fresh clean compilation. Ordinary compiler failures are not classified as clean-fallback requests.

Tracking issue: #132977

Implementation

Captures the effective IL provider chain, including generated P/Invoke stubs.
Validates all update assemblies before baseline object emission.
Resets the mutable MethodCodeNode code, GC, EH, debug, local, and dependency state required for recompilation.
Recompiles the union of methods changed by the current and previous updates, so sequential edits and reverts always derive from the original baseline.
Verifies ordered static and conditional dependencies, reasons, marked state, GC info, frame/unwind data, EH state, debug state, symbols, relocations/addends, alignment, COMDAT state, and object locations after code generation.
Records selected COFF fragment locations per compilation and binds the baseline assembly, configuration, and object with SHA-256.
Copies from the same verified baseline handle, validates every non-relocation byte, stages unique same-directory files, flushes them to disk, and publishes without overwrite.
Poisons retained state after any post-mutation failure so a failed attempt cannot be reused.

Supported envelope

Incremental compilation is accepted only for:

A Windows host targeting Windows x64 NativeAOT COFF, single-file compilation, and one primary input.
OptimizationMode.None with exactly one compiler thread.
Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.
Baseline and updated PE files with equal length, MVID, metadata method count, and complete non-body content after masking only timestamp, checksum, debug-directory, strong-name payload, and encoded method-body ranges.
Method bodies with unchanged encoded size, max stack, init-locals, local signature, and EH shape.
Non-constructor, non-generic leaf methods with exactly one marked, non-canonical, non-unboxing, non-foldable MethodCodeNode, no conditional dependencies or EH, and overlayable ECMA IL.
Identical allowed opcode streams where only explicit integer or floating-point constant operands change.
Selected non-COMDAT COFF fragments whose size, alignment, symbols, relocations, addends, GC/frame/EH/debug state, location, bounds, and overlap checks remain valid.

All other cases request a clean compilation.

RDM build-performance impact

Measured on one eligible edit to reachable Program.BuildModuleViewIndex in the RDM RdmNativeAotFast=true workload:

RDM step	Clean path	Retained incremental path	Impact
ILC object generation/update	640.395 s	0.291837 s	2,194.36x faster; 99.9544% less wall time
Native link	36.55 s	36.55 s	unchanged
ILC + native link developer loop	676.945 s (11m 16.9s)	36.842 s	18.37x faster; 94.56% less wall time
Time saved per eligible edit/link iteration	—	640.103 s (10m 40.1s)	derived from the rows above

The 3,744,339,247-byte incremental object exactly matched the clean object with SHA-256 B3140045782498DC4A06F712C2DAA329B732D6340DFCD3D80AD4181E17844206. The update reused 13,455,307 of 13,455,308 object nodes, patched one byte, and allocated 1,568,736 managed bytes.

These are measured component timings. A separate clean RDM publish measured 951.9 s (15m 51.9s), but a comparable complete incremental dotnet publish was not timed. The defensible practical result is the ~36.8-second edit-and-link loop instead of ~10–11 minutes, not a 36.8-second full build.

The first request still pays for the clean compilation and retains roughly 34–36 GiB. The result applies only to edits that pass the narrow safety gate.

Validation

build.cmd clr+libs+host baseline: succeeded with 0 warnings/errors.
build.cmd clr.aot+libs -rc Release -lc Release: succeeded with 0 warnings/errors.
ILCompiler.Compiler.Tests Release: 80 passed, 0 failed, 0 skipped.
Focused IncrementalCompilationTests: 58 passed, 0 failed, 0 skipped.
Dedicated priority-0 Windows-x64 NativeAOT incremental smoke test: runs automatically during BuildNativeAot and succeeded with 0 warnings/errors.
- Incremental edited object and independent clean edited object: 0BE481A1B058F826D4FCD0E013DEFC544BF4382E16CE8C5549EF94AE1F73666F.
- Reverted object and baseline object: 0CB70EB6CAA77ABC4C6F120AE64C1AF3F6370A37AEC4FD54FBF8A96179E44497.
- Explicit exit-85 rejection logging was validated under Windows PowerShell; CFG legs skip the unsupported differential path.
- Build-only comparison objects are written under the intermediate tree and excluded from Helix payloads.

Limitations

Normal compiler-driven source edits commonly change the MVID and therefore request a clean compilation. Productization would require a supported command/API contract, complete cross-process content keys for references/resources/toolchain/JIT/environment, request isolation, eviction and crash recovery, broader invalidation for optimized/inlined/preinitialized/reflection/generic/global facts, discovery of newly dirty nodes, validation or regeneration of linker-affecting side outputs, and compile-time-checked internal seams across the compiler assemblies.

Note

This pull request description was generated with GitHub Copilot.

azure-pipelines · 2026-08-31T06:18:00Z

Azure Pipelines: Successfully started running 4 pipeline(s). 12 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

dotnet-policy-service · 2026-08-31T06:18:41Z

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

jkotas · 2026-08-31T15:08:27Z

What are you trying to achieve with this?

It would be better to start with an issue discussing the experience you would like to see.

jkoritzinsky · 2026-08-31T15:52:03Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

awakecoding · 2026-08-31T16:11:11Z

What are you trying to achieve with this?
It would be better to start with an issue discussing the experience you would like to see.

I am trying to improve incremental build performance for NativeAOT on very large projects like Remote Desktop Manager. It currently takes a good 15 minutes to make a change and rebuild, and memory peaks over 32GB in ILC. I've been told you're supposed to just make managed builds with the warnings for NativeAOT to figure out what to fix for NativeAOT and then just wait on the NativeAOT build, trusting that it works, but this prevents actually trying out the NativeAOT build and repeatedly iterate on it. I know my project (RDM) is huge, but that doesn't mean we can't optimize the build tooling to introduce incremental builds

awakecoding · 2026-08-31T16:33:46Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

Where is this documented?

jkoritzinsky · 2026-08-31T16:50:04Z

You can set the IlcMultiModule property to true to try it out. Once again, no guarantees, this is unsupported, etc.

I also don't know if we actually have the MSBuild targets set up correctly to make this run in an incremental manner so this may need more work for that front.

awakecoding · 2026-08-31T18:21:15Z

@jkotas @jkoritzinsky I have created an issue here for NativeAOT incremental compilation support: #132977

awakecoding · 2026-08-31T20:11:03Z

Thanks — I tried IlcMultiModule on our RDM NativeAOT build. After working around a Windows target-plumbing issue (TargetOS was not forwarded to BuildFrameworkNativeObjects.proj), framework caching did work: 175 framework objects plus Framework.lib were byte/mtime-identical, and an identical publish dropped from 420.7s to 63.1s.

The app targets only compiled the top-level module, not its 509 non-framework references, so the stock link failed with 168 unresolved externals. I also tried custom per-module orchestration: 505/509 modules compiled and all 505 were reused on a warm run, but four modules hit rooting/generic issues and the aggregate link still failed with 1,248 unresolved externals. So this validates framework-level incrementality, but app-module orchestration and multifile generic/reflection support need more work before it is usable for this workload.

Note

This comment was drafted with GitHub Copilot.

MichalStrehovsky · 2026-08-31T22:10:20Z

Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.

This is just the start. An incremental compilation needs to disable all whole program optimizations, including trimming. These optimizations all have butterfly effects that are difficult to capture and reconstruct., A small change in one method might invalidate an optimization done elsewhere.

Once all the applicable optimizations are disabled, an incremental compilation mode gives:

Output that behaves differently from a real optimized native AOT compilation (this applies in the presence of trim/AOT unsafe code - the trimming+optimizations are invisible to trim/AOT safe code)
Subpar quality outputs because all the ways that native AOT could do better through trimming and whole program optimizations are gone.

If the problem is that the RDM codebase has many trimming warnings (I saw Newtonsoft) and instead of fixing the warnings you're rooting assemblies and retesting, you're not going to have a good time with native AOT or trimming and incremental compilation will not help you, unless you intend to ship the unoptimized incremental build.

First step to native AOT conversion is eliminating all the trimming warnings, we document that in the first section. Maybe the doc doesn't explain the gravity of the situation well enough but the "ensure there are no behavior changes by thoroughly testing your app after building as Native AOT" part really means that if there is any trimming warning, you need to retest the entire app after any change due to the butterfly effects. It is not possible to reliably ship a large app that has trimming warnings.

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

awakecoding · 2026-09-01T14:07:53Z

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

I understand where you are going with this, but I respectfully disagree: this means you have to fully port your app for NativeAOT before you can start trying it out despite the warnings. It's important to be able to iterate on the partial port early on to help with prioritization of the work to be done, but also early validation of code paths which differ significantly in a NativeAOT build as opposed to the managed build.

AI made it possible to consider porting RDM to NativeAOT, but it is still a moonshot project. The first blocker we recently resolved was porting all of RDM Windows from DevExpress+WinForms to Avalonia UI, which involved about 1500+ UI components. WinForms is fine for NativeAOT, but DevExpress isn't. That alone took almost a year with AI, and once finally fixed, I could have my first successful launches of the RDM Windows application with NativeAOT despite a ton of build warnings. Not all features work, but it's enough to start iterating and validating by prioritizing what areas of the application should be fixed next for NativeAOT safety.

The second major blocker we have resolved recently is the PowerShell SDK, which is in-process, and will always rely on JIT for obvious reasons. I didn't want to move things out of process because we expose .NET live objects in scripts in-process for some advanced features. I developed my own PowerShell SDK that remains in-process, and offers live object proxies from a NativeAOT-safe .NET application. Here's the custom PowerShell SDK I developed specifically for NativeAOT consumption: https://github.com/Devolutions/multi-pwsh

My point is not that I would like to ship RDM in production with tons of trim-safety warnings remaining. It is that I would like to iterate much faster on experimental (or production) NativeAOT builds. Managed builds differ too much from NativeAOT builds for them to be useful beyond producing warnings. For instance, my managed build of RDM uses the regular PowerShell SDK while the NativeAOT build uses my NativeAOT-safe PowerShell SDK replacement. Several features are also gated for NativeAOT-safety, as I work my way through the huge backlog of things to fix.

But let's say we fast forward to the future at a point where I have fixed all the NativeAOT build warnings. Even then, my NativeAOT builds will still be extremely slow (RDM won't magically become thin and lightweight), so it means I still have to use managed builds for the inner loop. Those managed builds will always significantly differ from a NativeAOT build. There is a point to be made about introducing incremental builds in NativeAOT in places where it can be done.

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds. I want to explore ways we can do it in places where there's a potential gain to be made, and then work on making the non-reusable parts of the build process faster, in hope that the NativeAOT build time becomes manageable. Right now, it's slow even for a CI build, and we need to bring it down.

agocke · 2026-09-01T14:42:46Z

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds.

I don't see how this follows. As Michal mentioned, whole program optimization means that a change in any method can produce changes in completely unrelated methods. For an incremental compilation to be correct this would somehow need to be fully accounted for, and it might end up leading to zero reuse in common cases.

agocke · 2026-09-01T16:45:13Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

awakecoding · 2026-09-01T17:32:18Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

thanks Andy, that explains it much better. I suggest closing the current PR, and keep the discussion going in the high-level NativeAOT incremental build issue: #132977

I would then research how we could take inspiration from rustc internals and come up with a new prototype for incremental builds in .NET NativeAOT does that doesn't disable certain passes. It would obviously be a lot more work, but I'm up for giving it a try.

MichalPetryka · 2026-09-01T17:59:40Z

The only prior art I'm aware of here is rustc.

MSVC does have incremental builds and LTCG, afair they track everything to handle wpo changes.

awakecoding · 2026-08-31T06:17:14Z

Summary

Adds a disabled-by-default, internal NativeAOT incremental-compilation prototype for Windows x64 COFF. A single ILC process retains its completed dependency graph, recompiles a prevalidated finite set of changed method bodies, and creates each updated object by patching a copy of the immutable baseline object.

The prototype adds no public API. Incremental rejection is explicit: ILC prints ILC_INCREMENTAL_REJECTED, exits with code 85, removes outputs it created, and requires the caller to start a fresh clean compilation. Ordinary compiler failures are not classified as clean-fallback requests.

Tracking issue: #132977

Implementation

Captures the effective IL provider chain, including generated P/Invoke stubs.
Validates all update assemblies before baseline object emission.
Resets the mutable MethodCodeNode code, GC, EH, debug, local, and dependency state required for recompilation.
Recompiles the union of methods changed by the current and previous updates, so sequential edits and reverts always derive from the original baseline.
Verifies ordered static and conditional dependencies, reasons, marked state, GC info, frame/unwind data, EH state, debug state, symbols, relocations/addends, alignment, COMDAT state, and object locations after code generation.
Records selected COFF fragment locations per compilation and binds the baseline assembly, configuration, and object with SHA-256.
Copies from the same verified baseline handle, validates every non-relocation byte, stages unique same-directory files, flushes them to disk, and publishes without overwrite.
Poisons retained state after any post-mutation failure so a failed attempt cannot be reused.

Supported envelope

Incremental compilation is accepted only for:

A Windows host targeting Windows x64 NativeAOT COFF, single-file compilation, and one primary input.
OptimizationMode.None with exactly one compiler thread.
Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.
Baseline and updated PE files with equal length, MVID, metadata method count, and complete non-body content after masking only timestamp, checksum, debug-directory, strong-name payload, and encoded method-body ranges.
Method bodies with unchanged encoded size, max stack, init-locals, local signature, and EH shape.
Non-constructor, non-generic leaf methods with exactly one marked, non-canonical, non-unboxing, non-foldable MethodCodeNode, no conditional dependencies or EH, and overlayable ECMA IL.
Identical allowed opcode streams where only explicit integer or floating-point constant operands change.
Selected non-COMDAT COFF fragments whose size, alignment, symbols, relocations, addends, GC/frame/EH/debug state, location, bounds, and overlap checks remain valid.

All other cases request a clean compilation.

RDM build-performance impact

Measured on one eligible edit to reachable Program.BuildModuleViewIndex in the RDM RdmNativeAotFast=true workload:

RDM step	Clean path	Retained incremental path	Impact
ILC object generation/update	640.395 s	0.291837 s	2,194.36x faster; 99.9544% less wall time
Native link	36.55 s	36.55 s	unchanged
ILC + native link developer loop	676.945 s (11m 16.9s)	36.842 s	18.37x faster; 94.56% less wall time
Time saved per eligible edit/link iteration	—	640.103 s (10m 40.1s)	derived from the rows above

The 3,744,339,247-byte incremental object exactly matched the clean object with SHA-256 B3140045782498DC4A06F712C2DAA329B732D6340DFCD3D80AD4181E17844206. The update reused 13,455,307 of 13,455,308 object nodes, patched one byte, and allocated 1,568,736 managed bytes.

These are measured component timings. A separate clean RDM publish measured 951.9 s (15m 51.9s), but a comparable complete incremental dotnet publish was not timed. The defensible practical result is the ~36.8-second edit-and-link loop instead of ~10–11 minutes, not a 36.8-second full build.

The first request still pays for the clean compilation and retains roughly 34–36 GiB. The result applies only to edits that pass the narrow safety gate.

Validation

build.cmd clr+libs+host baseline: succeeded with 0 warnings/errors.
build.cmd clr.aot+libs -rc Release -lc Release: succeeded with 0 warnings/errors.
ILCompiler.Compiler.Tests Release: 80 passed, 0 failed, 0 skipped.
Focused IncrementalCompilationTests: 58 passed, 0 failed, 0 skipped.
Dedicated priority-0 Windows-x64 NativeAOT incremental smoke test: runs automatically during BuildNativeAot and succeeded with 0 warnings/errors.
- Incremental edited object and independent clean edited object: 0BE481A1B058F826D4FCD0E013DEFC544BF4382E16CE8C5549EF94AE1F73666F.
- Reverted object and baseline object: 0CB70EB6CAA77ABC4C6F120AE64C1AF3F6370A37AEC4FD54FBF8A96179E44497.
- Explicit exit-85 rejection logging was validated under Windows PowerShell; CFG legs skip the unsupported differential path.
- Build-only comparison objects are written under the intermediate tree and excluded from Helix payloads.

Limitations

Normal compiler-driven source edits commonly change the MVID and therefore request a clean compilation. Productization would require a supported command/API contract, complete cross-process content keys for references/resources/toolchain/JIT/environment, request isolation, eviction and crash recovery, broader invalidation for optimized/inlined/preinitialized/reflection/generic/global facts, discovery of newly dirty nodes, validation or regeneration of linker-affecting side outputs, and compile-time-checked internal seams across the compiler assemblies.

Note

This pull request description was generated with GitHub Copilot.

azure-pipelines · 2026-08-31T06:18:00Z

Azure Pipelines: Successfully started running 4 pipeline(s). 12 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

dotnet-policy-service · 2026-08-31T06:18:41Z

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

jkotas · 2026-08-31T15:08:27Z

What are you trying to achieve with this?

It would be better to start with an issue discussing the experience you would like to see.

jkoritzinsky · 2026-08-31T15:52:03Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

awakecoding · 2026-08-31T16:11:11Z

What are you trying to achieve with this?
It would be better to start with an issue discussing the experience you would like to see.

I am trying to improve incremental build performance for NativeAOT on very large projects like Remote Desktop Manager. It currently takes a good 15 minutes to make a change and rebuild, and memory peaks over 32GB in ILC. I've been told you're supposed to just make managed builds with the warnings for NativeAOT to figure out what to fix for NativeAOT and then just wait on the NativeAOT build, trusting that it works, but this prevents actually trying out the NativeAOT build and repeatedly iterate on it. I know my project (RDM) is huge, but that doesn't mean we can't optimize the build tooling to introduce incremental builds

awakecoding · 2026-08-31T16:33:46Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

Where is this documented?

jkoritzinsky · 2026-08-31T16:50:04Z

You can set the IlcMultiModule property to true to try it out. Once again, no guarantees, this is unsupported, etc.

I also don't know if we actually have the MSBuild targets set up correctly to make this run in an incremental manner so this may need more work for that front.

awakecoding · 2026-08-31T18:21:15Z

@jkotas @jkoritzinsky I have created an issue here for NativeAOT incremental compilation support: #132977

awakecoding · 2026-08-31T20:11:03Z

Thanks — I tried IlcMultiModule on our RDM NativeAOT build. After working around a Windows target-plumbing issue (TargetOS was not forwarded to BuildFrameworkNativeObjects.proj), framework caching did work: 175 framework objects plus Framework.lib were byte/mtime-identical, and an identical publish dropped from 420.7s to 63.1s.

The app targets only compiled the top-level module, not its 509 non-framework references, so the stock link failed with 168 unresolved externals. I also tried custom per-module orchestration: 505/509 modules compiled and all 505 were reused on a warm run, but four modules hit rooting/generic issues and the aggregate link still failed with 1,248 unresolved externals. So this validates framework-level incrementality, but app-module orchestration and multifile generic/reflection support need more work before it is usable for this workload.

Note

This comment was drafted with GitHub Copilot.

MichalStrehovsky · 2026-08-31T22:10:20Z

Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.

This is just the start. An incremental compilation needs to disable all whole program optimizations, including trimming. These optimizations all have butterfly effects that are difficult to capture and reconstruct., A small change in one method might invalidate an optimization done elsewhere.

Once all the applicable optimizations are disabled, an incremental compilation mode gives:

Output that behaves differently from a real optimized native AOT compilation (this applies in the presence of trim/AOT unsafe code - the trimming+optimizations are invisible to trim/AOT safe code)
Subpar quality outputs because all the ways that native AOT could do better through trimming and whole program optimizations are gone.

If the problem is that the RDM codebase has many trimming warnings (I saw Newtonsoft) and instead of fixing the warnings you're rooting assemblies and retesting, you're not going to have a good time with native AOT or trimming and incremental compilation will not help you, unless you intend to ship the unoptimized incremental build.

First step to native AOT conversion is eliminating all the trimming warnings, we document that in the first section. Maybe the doc doesn't explain the gravity of the situation well enough but the "ensure there are no behavior changes by thoroughly testing your app after building as Native AOT" part really means that if there is any trimming warning, you need to retest the entire app after any change due to the butterfly effects. It is not possible to reliably ship a large app that has trimming warnings.

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

awakecoding · 2026-09-01T14:07:53Z

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

I understand where you are going with this, but I respectfully disagree: this means you have to fully port your app for NativeAOT before you can start trying it out despite the warnings. It's important to be able to iterate on the partial port early on to help with prioritization of the work to be done, but also early validation of code paths which differ significantly in a NativeAOT build as opposed to the managed build.

AI made it possible to consider porting RDM to NativeAOT, but it is still a moonshot project. The first blocker we recently resolved was porting all of RDM Windows from DevExpress+WinForms to Avalonia UI, which involved about 1500+ UI components. WinForms is fine for NativeAOT, but DevExpress isn't. That alone took almost a year with AI, and once finally fixed, I could have my first successful launches of the RDM Windows application with NativeAOT despite a ton of build warnings. Not all features work, but it's enough to start iterating and validating by prioritizing what areas of the application should be fixed next for NativeAOT safety.

The second major blocker we have resolved recently is the PowerShell SDK, which is in-process, and will always rely on JIT for obvious reasons. I didn't want to move things out of process because we expose .NET live objects in scripts in-process for some advanced features. I developed my own PowerShell SDK that remains in-process, and offers live object proxies from a NativeAOT-safe .NET application. Here's the custom PowerShell SDK I developed specifically for NativeAOT consumption: https://github.com/Devolutions/multi-pwsh

My point is not that I would like to ship RDM in production with tons of trim-safety warnings remaining. It is that I would like to iterate much faster on experimental (or production) NativeAOT builds. Managed builds differ too much from NativeAOT builds for them to be useful beyond producing warnings. For instance, my managed build of RDM uses the regular PowerShell SDK while the NativeAOT build uses my NativeAOT-safe PowerShell SDK replacement. Several features are also gated for NativeAOT-safety, as I work my way through the huge backlog of things to fix.

But let's say we fast forward to the future at a point where I have fixed all the NativeAOT build warnings. Even then, my NativeAOT builds will still be extremely slow (RDM won't magically become thin and lightweight), so it means I still have to use managed builds for the inner loop. Those managed builds will always significantly differ from a NativeAOT build. There is a point to be made about introducing incremental builds in NativeAOT in places where it can be done.

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds. I want to explore ways we can do it in places where there's a potential gain to be made, and then work on making the non-reusable parts of the build process faster, in hope that the NativeAOT build time becomes manageable. Right now, it's slow even for a CI build, and we need to bring it down.

agocke · 2026-09-01T14:42:46Z

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds.

I don't see how this follows. As Michal mentioned, whole program optimization means that a change in any method can produce changes in completely unrelated methods. For an incremental compilation to be correct this would somehow need to be fully accounted for, and it might end up leading to zero reuse in common cases.

agocke · 2026-09-01T16:45:13Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

awakecoding · 2026-09-01T17:32:18Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

thanks Andy, that explains it much better. I suggest closing the current PR, and keep the discussion going in the high-level NativeAOT incremental build issue: #132977

I would then research how we could take inspiration from rustc internals and come up with a new prototype for incremental builds in .NET NativeAOT does that doesn't disable certain passes. It would obviously be a lot more work, but I'm up for giving it a try.

MichalPetryka · 2026-09-01T17:59:40Z

The only prior art I'm aware of here is rustc.

MSVC does have incremental builds and LTCG, afair they track everything to handle wpo changes.

awakecoding · 2026-08-31T06:17:14Z

Summary

Adds a disabled-by-default, internal NativeAOT incremental-compilation prototype for Windows x64 COFF. A single ILC process retains its completed dependency graph, recompiles a prevalidated finite set of changed method bodies, and creates each updated object by patching a copy of the immutable baseline object.

The prototype adds no public API. Incremental rejection is explicit: ILC prints ILC_INCREMENTAL_REJECTED, exits with code 85, removes outputs it created, and requires the caller to start a fresh clean compilation. Ordinary compiler failures are not classified as clean-fallback requests.

Tracking issue: #132977

Implementation

Captures the effective IL provider chain, including generated P/Invoke stubs.
Validates all update assemblies before baseline object emission.
Resets the mutable MethodCodeNode code, GC, EH, debug, local, and dependency state required for recompilation.
Recompiles the union of methods changed by the current and previous updates, so sequential edits and reverts always derive from the original baseline.
Verifies ordered static and conditional dependencies, reasons, marked state, GC info, frame/unwind data, EH state, debug state, symbols, relocations/addends, alignment, COMDAT state, and object locations after code generation.
Records selected COFF fragment locations per compilation and binds the baseline assembly, configuration, and object with SHA-256.
Copies from the same verified baseline handle, validates every non-relocation byte, stages unique same-directory files, flushes them to disk, and publishes without overwrite.
Poisons retained state after any post-mutation failure so a failed attempt cannot be reused.

Supported envelope

Incremental compilation is accepted only for:

A Windows host targeting Windows x64 NativeAOT COFF, single-file compilation, and one primary input.
OptimizationMode.None with exactly one compiler thread.
Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.
Baseline and updated PE files with equal length, MVID, metadata method count, and complete non-body content after masking only timestamp, checksum, debug-directory, strong-name payload, and encoded method-body ranges.
Method bodies with unchanged encoded size, max stack, init-locals, local signature, and EH shape.
Non-constructor, non-generic leaf methods with exactly one marked, non-canonical, non-unboxing, non-foldable MethodCodeNode, no conditional dependencies or EH, and overlayable ECMA IL.
Identical allowed opcode streams where only explicit integer or floating-point constant operands change.
Selected non-COMDAT COFF fragments whose size, alignment, symbols, relocations, addends, GC/frame/EH/debug state, location, bounds, and overlap checks remain valid.

All other cases request a clean compilation.

RDM build-performance impact

Measured on one eligible edit to reachable Program.BuildModuleViewIndex in the RDM RdmNativeAotFast=true workload:

RDM step	Clean path	Retained incremental path	Impact
ILC object generation/update	640.395 s	0.291837 s	2,194.36x faster; 99.9544% less wall time
Native link	36.55 s	36.55 s	unchanged
ILC + native link developer loop	676.945 s (11m 16.9s)	36.842 s	18.37x faster; 94.56% less wall time
Time saved per eligible edit/link iteration	—	640.103 s (10m 40.1s)	derived from the rows above

The 3,744,339,247-byte incremental object exactly matched the clean object with SHA-256 B3140045782498DC4A06F712C2DAA329B732D6340DFCD3D80AD4181E17844206. The update reused 13,455,307 of 13,455,308 object nodes, patched one byte, and allocated 1,568,736 managed bytes.

These are measured component timings. A separate clean RDM publish measured 951.9 s (15m 51.9s), but a comparable complete incremental dotnet publish was not timed. The defensible practical result is the ~36.8-second edit-and-link loop instead of ~10–11 minutes, not a 36.8-second full build.

The first request still pays for the clean compilation and retains roughly 34–36 GiB. The result applies only to edits that pass the narrow safety gate.

Validation

build.cmd clr+libs+host baseline: succeeded with 0 warnings/errors.
build.cmd clr.aot+libs -rc Release -lc Release: succeeded with 0 warnings/errors.
ILCompiler.Compiler.Tests Release: 80 passed, 0 failed, 0 skipped.
Focused IncrementalCompilationTests: 58 passed, 0 failed, 0 skipped.
Dedicated priority-0 Windows-x64 NativeAOT incremental smoke test: runs automatically during BuildNativeAot and succeeded with 0 warnings/errors.
- Incremental edited object and independent clean edited object: 0BE481A1B058F826D4FCD0E013DEFC544BF4382E16CE8C5549EF94AE1F73666F.
- Reverted object and baseline object: 0CB70EB6CAA77ABC4C6F120AE64C1AF3F6370A37AEC4FD54FBF8A96179E44497.
- Explicit exit-85 rejection logging was validated under Windows PowerShell; CFG legs skip the unsupported differential path.
- Build-only comparison objects are written under the intermediate tree and excluded from Helix payloads.

Limitations

Normal compiler-driven source edits commonly change the MVID and therefore request a clean compilation. Productization would require a supported command/API contract, complete cross-process content keys for references/resources/toolchain/JIT/environment, request isolation, eviction and crash recovery, broader invalidation for optimized/inlined/preinitialized/reflection/generic/global facts, discovery of newly dirty nodes, validation or regeneration of linker-affecting side outputs, and compile-time-checked internal seams across the compiler assemblies.

Note

This pull request description was generated with GitHub Copilot.

azure-pipelines · 2026-08-31T06:18:00Z

Azure Pipelines: Successfully started running 4 pipeline(s). 12 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

dotnet-policy-service · 2026-08-31T06:18:41Z

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

jkotas · 2026-08-31T15:08:27Z

What are you trying to achieve with this?

It would be better to start with an issue discussing the experience you would like to see.

jkoritzinsky · 2026-08-31T15:52:03Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

awakecoding · 2026-08-31T16:11:11Z

What are you trying to achieve with this?
It would be better to start with an issue discussing the experience you would like to see.

I am trying to improve incremental build performance for NativeAOT on very large projects like Remote Desktop Manager. It currently takes a good 15 minutes to make a change and rebuild, and memory peaks over 32GB in ILC. I've been told you're supposed to just make managed builds with the warnings for NativeAOT to figure out what to fix for NativeAOT and then just wait on the NativeAOT build, trusting that it works, but this prevents actually trying out the NativeAOT build and repeatedly iterate on it. I know my project (RDM) is huge, but that doesn't mean we can't optimize the build tooling to introduce incremental builds

awakecoding · 2026-08-31T16:33:46Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

Where is this documented?

jkoritzinsky · 2026-08-31T16:50:04Z

You can set the IlcMultiModule property to true to try it out. Once again, no guarantees, this is unsupported, etc.

I also don't know if we actually have the MSBuild targets set up correctly to make this run in an incremental manner so this may need more work for that front.

awakecoding · 2026-08-31T18:21:15Z

@jkotas @jkoritzinsky I have created an issue here for NativeAOT incremental compilation support: #132977

awakecoding · 2026-08-31T20:11:03Z

Thanks — I tried IlcMultiModule on our RDM NativeAOT build. After working around a Windows target-plumbing issue (TargetOS was not forwarded to BuildFrameworkNativeObjects.proj), framework caching did work: 175 framework objects plus Framework.lib were byte/mtime-identical, and an identical publish dropped from 420.7s to 63.1s.

The app targets only compiled the top-level module, not its 509 non-framework references, so the stock link failed with 168 unresolved externals. I also tried custom per-module orchestration: 505/509 modules compiled and all 505 were reused on a warm run, but four modules hit rooting/generic issues and the aggregate link still failed with 1,248 unresolved externals. So this validates framework-level incrementality, but app-module orchestration and multifile generic/reflection support need more work before it is usable for this workload.

Note

This comment was drafted with GitHub Copilot.

MichalStrehovsky · 2026-08-31T22:10:20Z

Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.

This is just the start. An incremental compilation needs to disable all whole program optimizations, including trimming. These optimizations all have butterfly effects that are difficult to capture and reconstruct., A small change in one method might invalidate an optimization done elsewhere.

Once all the applicable optimizations are disabled, an incremental compilation mode gives:

Output that behaves differently from a real optimized native AOT compilation (this applies in the presence of trim/AOT unsafe code - the trimming+optimizations are invisible to trim/AOT safe code)
Subpar quality outputs because all the ways that native AOT could do better through trimming and whole program optimizations are gone.

If the problem is that the RDM codebase has many trimming warnings (I saw Newtonsoft) and instead of fixing the warnings you're rooting assemblies and retesting, you're not going to have a good time with native AOT or trimming and incremental compilation will not help you, unless you intend to ship the unoptimized incremental build.

First step to native AOT conversion is eliminating all the trimming warnings, we document that in the first section. Maybe the doc doesn't explain the gravity of the situation well enough but the "ensure there are no behavior changes by thoroughly testing your app after building as Native AOT" part really means that if there is any trimming warning, you need to retest the entire app after any change due to the butterfly effects. It is not possible to reliably ship a large app that has trimming warnings.

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

awakecoding · 2026-09-01T14:07:53Z

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

I understand where you are going with this, but I respectfully disagree: this means you have to fully port your app for NativeAOT before you can start trying it out despite the warnings. It's important to be able to iterate on the partial port early on to help with prioritization of the work to be done, but also early validation of code paths which differ significantly in a NativeAOT build as opposed to the managed build.

AI made it possible to consider porting RDM to NativeAOT, but it is still a moonshot project. The first blocker we recently resolved was porting all of RDM Windows from DevExpress+WinForms to Avalonia UI, which involved about 1500+ UI components. WinForms is fine for NativeAOT, but DevExpress isn't. That alone took almost a year with AI, and once finally fixed, I could have my first successful launches of the RDM Windows application with NativeAOT despite a ton of build warnings. Not all features work, but it's enough to start iterating and validating by prioritizing what areas of the application should be fixed next for NativeAOT safety.

The second major blocker we have resolved recently is the PowerShell SDK, which is in-process, and will always rely on JIT for obvious reasons. I didn't want to move things out of process because we expose .NET live objects in scripts in-process for some advanced features. I developed my own PowerShell SDK that remains in-process, and offers live object proxies from a NativeAOT-safe .NET application. Here's the custom PowerShell SDK I developed specifically for NativeAOT consumption: https://github.com/Devolutions/multi-pwsh

My point is not that I would like to ship RDM in production with tons of trim-safety warnings remaining. It is that I would like to iterate much faster on experimental (or production) NativeAOT builds. Managed builds differ too much from NativeAOT builds for them to be useful beyond producing warnings. For instance, my managed build of RDM uses the regular PowerShell SDK while the NativeAOT build uses my NativeAOT-safe PowerShell SDK replacement. Several features are also gated for NativeAOT-safety, as I work my way through the huge backlog of things to fix.

But let's say we fast forward to the future at a point where I have fixed all the NativeAOT build warnings. Even then, my NativeAOT builds will still be extremely slow (RDM won't magically become thin and lightweight), so it means I still have to use managed builds for the inner loop. Those managed builds will always significantly differ from a NativeAOT build. There is a point to be made about introducing incremental builds in NativeAOT in places where it can be done.

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds. I want to explore ways we can do it in places where there's a potential gain to be made, and then work on making the non-reusable parts of the build process faster, in hope that the NativeAOT build time becomes manageable. Right now, it's slow even for a CI build, and we need to bring it down.

agocke · 2026-09-01T14:42:46Z

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds.

I don't see how this follows. As Michal mentioned, whole program optimization means that a change in any method can produce changes in completely unrelated methods. For an incremental compilation to be correct this would somehow need to be fully accounted for, and it might end up leading to zero reuse in common cases.

agocke · 2026-09-01T16:45:13Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

awakecoding · 2026-09-01T17:32:18Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

thanks Andy, that explains it much better. I suggest closing the current PR, and keep the discussion going in the high-level NativeAOT incremental build issue: #132977

I would then research how we could take inspiration from rustc internals and come up with a new prototype for incremental builds in .NET NativeAOT does that doesn't disable certain passes. It would obviously be a lot more work, but I'm up for giving it a try.

MichalPetryka · 2026-09-01T17:59:40Z

The only prior art I'm aware of here is rustc.

MSVC does have incremental builds and LTCG, afair they track everything to handle wpo changes.

awakecoding · 2026-08-31T06:17:14Z

Summary

Adds a disabled-by-default, internal NativeAOT incremental-compilation prototype for Windows x64 COFF. A single ILC process retains its completed dependency graph, recompiles a prevalidated finite set of changed method bodies, and creates each updated object by patching a copy of the immutable baseline object.

The prototype adds no public API. Incremental rejection is explicit: ILC prints ILC_INCREMENTAL_REJECTED, exits with code 85, removes outputs it created, and requires the caller to start a fresh clean compilation. Ordinary compiler failures are not classified as clean-fallback requests.

Tracking issue: #132977

Implementation

Captures the effective IL provider chain, including generated P/Invoke stubs.
Validates all update assemblies before baseline object emission.
Resets the mutable MethodCodeNode code, GC, EH, debug, local, and dependency state required for recompilation.
Recompiles the union of methods changed by the current and previous updates, so sequential edits and reverts always derive from the original baseline.
Verifies ordered static and conditional dependencies, reasons, marked state, GC info, frame/unwind data, EH state, debug state, symbols, relocations/addends, alignment, COMDAT state, and object locations after code generation.
Records selected COFF fragment locations per compilation and binds the baseline assembly, configuration, and object with SHA-256.
Copies from the same verified baseline handle, validates every non-relocation byte, stages unique same-directory files, flushes them to disk, and publishes without overwrite.
Poisons retained state after any post-mutation failure so a failed attempt cannot be reused.

Supported envelope

Incremental compilation is accepted only for:

A Windows host targeting Windows x64 NativeAOT COFF, single-file compilation, and one primary input.
OptimizationMode.None with exactly one compiler thread.
Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.
Baseline and updated PE files with equal length, MVID, metadata method count, and complete non-body content after masking only timestamp, checksum, debug-directory, strong-name payload, and encoded method-body ranges.
Method bodies with unchanged encoded size, max stack, init-locals, local signature, and EH shape.
Non-constructor, non-generic leaf methods with exactly one marked, non-canonical, non-unboxing, non-foldable MethodCodeNode, no conditional dependencies or EH, and overlayable ECMA IL.
Identical allowed opcode streams where only explicit integer or floating-point constant operands change.
Selected non-COMDAT COFF fragments whose size, alignment, symbols, relocations, addends, GC/frame/EH/debug state, location, bounds, and overlap checks remain valid.

All other cases request a clean compilation.

RDM build-performance impact

Measured on one eligible edit to reachable Program.BuildModuleViewIndex in the RDM RdmNativeAotFast=true workload:

RDM step	Clean path	Retained incremental path	Impact
ILC object generation/update	640.395 s	0.291837 s	2,194.36x faster; 99.9544% less wall time
Native link	36.55 s	36.55 s	unchanged
ILC + native link developer loop	676.945 s (11m 16.9s)	36.842 s	18.37x faster; 94.56% less wall time
Time saved per eligible edit/link iteration	—	640.103 s (10m 40.1s)	derived from the rows above

The 3,744,339,247-byte incremental object exactly matched the clean object with SHA-256 B3140045782498DC4A06F712C2DAA329B732D6340DFCD3D80AD4181E17844206. The update reused 13,455,307 of 13,455,308 object nodes, patched one byte, and allocated 1,568,736 managed bytes.

These are measured component timings. A separate clean RDM publish measured 951.9 s (15m 51.9s), but a comparable complete incremental dotnet publish was not timed. The defensible practical result is the ~36.8-second edit-and-link loop instead of ~10–11 minutes, not a 36.8-second full build.

The first request still pays for the clean compilation and retains roughly 34–36 GiB. The result applies only to edits that pass the narrow safety gate.

Validation

build.cmd clr+libs+host baseline: succeeded with 0 warnings/errors.
build.cmd clr.aot+libs -rc Release -lc Release: succeeded with 0 warnings/errors.
ILCompiler.Compiler.Tests Release: 80 passed, 0 failed, 0 skipped.
Focused IncrementalCompilationTests: 58 passed, 0 failed, 0 skipped.
Dedicated priority-0 Windows-x64 NativeAOT incremental smoke test: runs automatically during BuildNativeAot and succeeded with 0 warnings/errors.
- Incremental edited object and independent clean edited object: 0BE481A1B058F826D4FCD0E013DEFC544BF4382E16CE8C5549EF94AE1F73666F.
- Reverted object and baseline object: 0CB70EB6CAA77ABC4C6F120AE64C1AF3F6370A37AEC4FD54FBF8A96179E44497.
- Explicit exit-85 rejection logging was validated under Windows PowerShell; CFG legs skip the unsupported differential path.
- Build-only comparison objects are written under the intermediate tree and excluded from Helix payloads.

Limitations

Normal compiler-driven source edits commonly change the MVID and therefore request a clean compilation. Productization would require a supported command/API contract, complete cross-process content keys for references/resources/toolchain/JIT/environment, request isolation, eviction and crash recovery, broader invalidation for optimized/inlined/preinitialized/reflection/generic/global facts, discovery of newly dirty nodes, validation or regeneration of linker-affecting side outputs, and compile-time-checked internal seams across the compiler assemblies.

Note

This pull request description was generated with GitHub Copilot.

azure-pipelines · 2026-08-31T06:18:00Z

Azure Pipelines: Successfully started running 4 pipeline(s). 12 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

dotnet-policy-service · 2026-08-31T06:18:41Z

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

jkotas · 2026-08-31T15:08:27Z

What are you trying to achieve with this?

It would be better to start with an issue discussing the experience you would like to see.

jkoritzinsky · 2026-08-31T15:52:03Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

awakecoding · 2026-08-31T16:11:11Z

What are you trying to achieve with this?
It would be better to start with an issue discussing the experience you would like to see.

I am trying to improve incremental build performance for NativeAOT on very large projects like Remote Desktop Manager. It currently takes a good 15 minutes to make a change and rebuild, and memory peaks over 32GB in ILC. I've been told you're supposed to just make managed builds with the warnings for NativeAOT to figure out what to fix for NativeAOT and then just wait on the NativeAOT build, trusting that it works, but this prevents actually trying out the NativeAOT build and repeatedly iterate on it. I know my project (RDM) is huge, but that doesn't mean we can't optimize the build tooling to introduce incremental builds

awakecoding · 2026-08-31T16:33:46Z

I feel like the existing "multifile" mode that we haven't productized is a better approach here (one object file per assembly, would allow incremental ILC execution at the assembly boundary).

Where is this documented?

jkoritzinsky · 2026-08-31T16:50:04Z

You can set the IlcMultiModule property to true to try it out. Once again, no guarantees, this is unsupported, etc.

I also don't know if we actually have the MSBuild targets set up correctly to make this run in an incremental manner so this may need more work for that front.

awakecoding · 2026-08-31T18:21:15Z

@jkotas @jkoritzinsky I have created an issue here for NativeAOT incremental compilation support: #132977

awakecoding · 2026-08-31T20:11:03Z

Thanks — I tried IlcMultiModule on our RDM NativeAOT build. After working around a Windows target-plumbing issue (TargetOS was not forwarded to BuildFrameworkNativeObjects.proj), framework caching did work: 175 framework objects plus Framework.lib were byte/mtime-identical, and an identical publish dropped from 420.7s to 63.1s.

The app targets only compiled the top-level module, not its 509 non-framework references, so the stock link failed with 168 unresolved externals. I also tried custom per-module orchestration: 505/509 modules compiled and all 505 were reused on a warm run, but four modules hit rooting/generic issues and the aggregate link still failed with 1,248 unresolved externals. So this validates framework-level incrementality, but app-module orchestration and multifile generic/reflection support need more work before it is usable for this workload.

Note

This comment was drafted with GitHub Copilot.

MichalStrehovsky · 2026-08-31T22:10:20Z

Scanner, preinitialization, custom inlining, method folding, native debug info, profile/order layout, custom JIT configuration, CFG, resilience, dehydration, dependency logs, exports, maps, metadata/SourceLink output, reachability modes, and other side outputs disabled.

This is just the start. An incremental compilation needs to disable all whole program optimizations, including trimming. These optimizations all have butterfly effects that are difficult to capture and reconstruct., A small change in one method might invalidate an optimization done elsewhere.

Once all the applicable optimizations are disabled, an incremental compilation mode gives:

Output that behaves differently from a real optimized native AOT compilation (this applies in the presence of trim/AOT unsafe code - the trimming+optimizations are invisible to trim/AOT safe code)
Subpar quality outputs because all the ways that native AOT could do better through trimming and whole program optimizations are gone.

If the problem is that the RDM codebase has many trimming warnings (I saw Newtonsoft) and instead of fixing the warnings you're rooting assemblies and retesting, you're not going to have a good time with native AOT or trimming and incremental compilation will not help you, unless you intend to ship the unoptimized incremental build.

First step to native AOT conversion is eliminating all the trimming warnings, we document that in the first section. Maybe the doc doesn't explain the gravity of the situation well enough but the "ensure there are no behavior changes by thoroughly testing your app after building as Native AOT" part really means that if there is any trimming warning, you need to retest the entire app after any change due to the butterfly effects. It is not possible to reliably ship a large app that has trimming warnings.

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

awakecoding · 2026-09-01T14:07:53Z

It would be better use of your tokens to have AI replace all the trim unsafe code in RDM. I bet the compile times will get more reasonable too once assemblies are no longer reflection-rooted.

I understand where you are going with this, but I respectfully disagree: this means you have to fully port your app for NativeAOT before you can start trying it out despite the warnings. It's important to be able to iterate on the partial port early on to help with prioritization of the work to be done, but also early validation of code paths which differ significantly in a NativeAOT build as opposed to the managed build.

AI made it possible to consider porting RDM to NativeAOT, but it is still a moonshot project. The first blocker we recently resolved was porting all of RDM Windows from DevExpress+WinForms to Avalonia UI, which involved about 1500+ UI components. WinForms is fine for NativeAOT, but DevExpress isn't. That alone took almost a year with AI, and once finally fixed, I could have my first successful launches of the RDM Windows application with NativeAOT despite a ton of build warnings. Not all features work, but it's enough to start iterating and validating by prioritizing what areas of the application should be fixed next for NativeAOT safety.

The second major blocker we have resolved recently is the PowerShell SDK, which is in-process, and will always rely on JIT for obvious reasons. I didn't want to move things out of process because we expose .NET live objects in scripts in-process for some advanced features. I developed my own PowerShell SDK that remains in-process, and offers live object proxies from a NativeAOT-safe .NET application. Here's the custom PowerShell SDK I developed specifically for NativeAOT consumption: https://github.com/Devolutions/multi-pwsh

My point is not that I would like to ship RDM in production with tons of trim-safety warnings remaining. It is that I would like to iterate much faster on experimental (or production) NativeAOT builds. Managed builds differ too much from NativeAOT builds for them to be useful beyond producing warnings. For instance, my managed build of RDM uses the regular PowerShell SDK while the NativeAOT build uses my NativeAOT-safe PowerShell SDK replacement. Several features are also gated for NativeAOT-safety, as I work my way through the huge backlog of things to fix.

But let's say we fast forward to the future at a point where I have fixed all the NativeAOT build warnings. Even then, my NativeAOT builds will still be extremely slow (RDM won't magically become thin and lightweight), so it means I still have to use managed builds for the inner loop. Those managed builds will always significantly differ from a NativeAOT build. There is a point to be made about introducing incremental builds in NativeAOT in places where it can be done.

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds. I want to explore ways we can do it in places where there's a potential gain to be made, and then work on making the non-reusable parts of the build process faster, in hope that the NativeAOT build time becomes manageable. Right now, it's slow even for a CI build, and we need to bring it down.

agocke · 2026-09-01T14:42:46Z

I understand whole program optimization comes with its challenges - but that doesn't mean there is zero opportunity for introducing at least partial artifact reuse in between builds.

I don't see how this follows. As Michal mentioned, whole program optimization means that a change in any method can produce changes in completely unrelated methods. For an incremental compilation to be correct this would somehow need to be fully accounted for, and it might end up leading to zero reuse in common cases.

agocke · 2026-09-01T16:45:13Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

awakecoding · 2026-09-01T17:32:18Z

Thinking about this more, I think the fundamental problem with this PR is it tries to tackle the problem by disabling certain passes, which puts everything in an unsupported state. The proper path to incremental support probably goes through incremental dependency analysis. Even then, I'm not sure how much time you would actually save in practice. It's also a big project. The only prior art I'm aware of here is rustc.

thanks Andy, that explains it much better. I suggest closing the current PR, and keep the discussion going in the high-level NativeAOT incremental build issue: #132977

I would then research how we could take inspiration from rustc internals and come up with a new prototype for incremental builds in .NET NativeAOT does that doesn't disable certain passes. It would obviously be a lot more work, but I'm up for giving it a try.

MichalPetryka · 2026-09-01T17:59:40Z

The only prior art I'm aware of here is rustc.

MSVC does have incremental builds and LTCG, afair they track everything to handle wpo changes.

Prototype incremental NativeAOT compilation
82fcff7

github-actionsBot added the area-NativeAOT-coreclr label Aug 31, 2026