Skip to content

Convert managed ilasm to use ANTLR actions instead of visiting the parse tree - #132346

Open
jkoritzinsky wants to merge 19 commits into
ilasm-more-testsfrom
ilasm-actions
Open

Convert managed ilasm to use ANTLR actions instead of visiting the parse tree#132346
jkoritzinsky wants to merge 19 commits into
ilasm-more-testsfrom
ilasm-actions

Conversation

@jkoritzinsky

Copy link
Copy Markdown
Member

Currently, managed ilasm builds a parse tree for the whole input .il file. This causes massive memory allocations (and OOMs on x86) for some of our larger test assemblies (HardwareIntrinsics AVX512 and HugeArray) that are on the scale of CoreLib. It also causes a massive slowdown due to the allocations.

This PR converts our parse tree visitor to instead be called as separate "actions" during grammar parsing, similar to how native ilasm does in the YACC file. Unlike native ilasm, all of the real logic is in C# files, and the logic in the CIL.g4 file is minimal (function calls into C#).

Because ANTLR4 only emits public types, we limit the exposed public API surface for (expected) internal consumers of the ILAssembler library (such as Roslyn for their test tree) to only the expected public API surface.


Stack created with GitHub Stacks CLIGive Feedback 💬

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 5 pipeline(s).
11 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @JulieLeeMSFT, @dotnet/jit-contrib
See info in area-owners.md if you want to be subscribed.


```
./dotnet.sh build src/tools/ilasm/src/ILAssembler/gen
./dotnet.sh build src/tools/ilasm/src/ILAssembler

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intentional that gen was dropped here?

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request refactors the managed ILAssembler pipeline to avoid building a full ANTLR parse tree by switching to action-driven parsing (with an unbuffered token stream), and introduces an explicit reference-assembly “contract” to constrain the intended public API surface while keeping ANTLR-generated types out of the supported contract.

Changes:

  • Switch DocumentCompiler to parse with UnbufferedTokenStream, disable parse-tree construction, and drive compilation through GrammarActions.
  • Add a custom reference-assembly project under src/tools/ilasm/src/ILAssembler/ref/ and adjust tests to compile against implementation where needed.
  • Add extensive regression tests for error-tolerant parsing and state-leak prevention across documents/scopes.

Reviewed changes

Copilot reviewed 83 out of 88 changed files in this pull request and generated 2 comments.

Show a summary per file
FileDescription
src/tools/ilasm/tests/ILAssembler.Tests/TypedefTests.csUpdates custom-attribute assertions to match new emission/binding behavior.
src/tools/ilasm/tests/ILAssembler.Tests/SyntaxTests.csAdds corpora for truncated/malformed input to ensure diagnostics are produced instead of exceptions.
src/tools/ilasm/tests/ILAssembler.Tests/SourceDirectiveTests.csAdds multi-document and malformed language directive tests to ensure debug state doesn’t leak.
src/tools/ilasm/tests/ILAssembler.Tests/SecurityTests.csAdds permissionset ordering/leak tests; expands coverage for structured security declarations.
src/tools/ilasm/tests/ILAssembler.Tests/MethodTests.csAdds tests for override ownership and state isolation after malformed nested method headers.
src/tools/ilasm/tests/ILAssembler.Tests/InteropTests.csAdds coverage for raw marshal blobs and “no state leak” scenarios across fields/documents.
src/tools/ilasm/tests/ILAssembler.Tests/ILAssembler.Tests.csprojOpts into implementation assembly for tests and links additional CLI helper code.
src/tools/ilasm/tests/ILAssembler.Tests/FieldTests.csAdds tests ensuring trailing .custom directives bind correctly and don’t leak across scopes.
src/tools/ilasm/tests/ILAssembler.Tests/ExceptionHandlingTests.csAdds ordering/bounds tests for EH region synthesis and catch-type resolution timing.
src/tools/ilasm/tests/ILAssembler.Tests/EventTests.csAdds test for .event without an explicit type emitting a nil event type.
src/tools/ilasm/tests/ILAssembler.Tests/DocumentCompilerTests.csAdds tests ensuring truncated/syntax-broken documents don’t leak namespace/type scopes into subsequent documents.
src/tools/ilasm/tests/ILAssembler.Tests/DataTests.csAdds large bytearray/data declaration tests and leak-prevention tests across documents.
src/tools/ilasm/tests/ILAssembler.Tests/CommandLineTests.csAdds a Windows sharing-violation retry test for output writing behavior.
src/tools/ilasm/tests/ILAssembler.Tests/AssemblyTests.csAdds coverage for accepting “legacy library” assembly attribute variants.
src/tools/ilasm/src/ILAssembler/TypeName.csRemoves the old internal TypeName record (migrated into the new semantic model).
src/tools/ilasm/src/ILAssembler/StringCharStream.csAdds a lightweight ICharStream over string to avoid AntlrInputStream overhead.
src/tools/ilasm/src/ILAssembler/ref/ILAssembler/SourceText.csAdds contract type for document text + path.
src/tools/ilasm/src/ILAssembler/ref/ILAssembler/SourceSpan.csAdds contract type for source spans.
src/tools/ilasm/src/ILAssembler/ref/ILAssembler/Options.csAdds contract type defining compiler options.
src/tools/ilasm/src/ILAssembler/ref/ILAssembler/Location.csAdds contract type for diagnostic locations.
src/tools/ilasm/src/ILAssembler/ref/ILAssembler/DocumentCompiler.csAdds contract surface for the compiler entry points.
src/tools/ilasm/src/ILAssembler/ref/ILAssembler/Diagnostic.csAdds contract surface for diagnostics and ids.
src/tools/ilasm/src/ILAssembler/ref/ILAssembler/CompilationResult.csAdds contract surface for serializing compiler output.
src/tools/ilasm/src/ILAssembler/ref/ILAssembler.csprojIntroduces the ref-contract project.
src/tools/ilasm/src/ILAssembler/PreprocessedTokenSource.csKeeps root source on stack at EOF to support parser error recovery on truncated input.
src/tools/ilasm/src/ILAssembler/ILAssembler.csprojHooks up the new contract project and excludes ref sources from implementation compile.
src/tools/ilasm/src/ILAssembler/gen/ilasm-generator.csprojDisables visitor generation and removes generated visitor/parser compile items from the generator build.
src/tools/ilasm/src/ILAssembler/EntityRegistry.csAllows events to have an absent type and emits a nil handle accordingly.
src/tools/ilasm/src/ILAssembler/DocumentCompiler.csSwitches from visitor-based traversal to action-driven parsing with parse-tree disabled.
src/tools/ilasm/src/ILAssembler/CompatibilitySuppressions.xmlAdds CP0001 suppressions for intentionally unsupported implementation-only public types.
src/tools/ilasm/src/ILAssembler/CILParser.SemanticValues.Signatures.csAdds strongly-typed semantic value model for signatures/types/member refs.
src/tools/ilasm/src/ILAssembler/CILParser.SemanticValues.MethodBodies.csAdds semantic value model for method bodies (EH/debug/data/security/instruction values).
src/tools/ilasm/src/ILAssembler/CILParser.SemanticValues.Marshalling.csAdds semantic value model for marshalling/native type synthesis.
src/tools/ilasm/src/ILAssembler/CILParser.SemanticValues.Manifest.csAdds semantic value model for manifest-level constructs (assembly/resources/typedef/etc.).
src/tools/ilasm/src/ILAssembler/CILParser.SemanticValues.Declarations.csAdds semantic value model for headers/builders used during action parsing.
src/tools/ilasm/src/ILAssembler/CILParser.SemanticValues.CustomAttributes.csAdds semantic value model for custom attributes and serialization blobs.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Types.References.csImplements action helpers for type-name construction and type resolution.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Types.Headers.Actions.csImplements action helpers for namespace/class header parsing and attribute synthesis.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Types.csImplements scope ownership/unwind to prevent state leaks across declarations.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Signatures.References.csImplements action helpers for signature/member reference synthesis and materialization.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Signatures.csProvides thin helpers connecting parser contexts to the materializers.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Security.csImplements action-driven declarative security parsing and permission-set emission.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.MethodHeaders.Generics.csImplements generic parameter name/constraint materialization logic.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.MethodHeaders.csImplements action-driven method definition creation/signature emission.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.MethodBodies.csAdds label validation and method-name parsing actions.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Members.Fields.csImplements action-driven field declaration parsing and emission; tracks pending .custom binding.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Manifest.VTable.csImplements vtfixup parsing; rejects raw .vtable blob support.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Manifest.Typedefs.csImplements typedef parsing and alias resolution materialization.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Manifest.Resources.csImplements action-driven manifest resource declaration materialization.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Manifest.References.csImplements action-driven assembly reference header/declarations materialization.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Manifest.Files.csImplements action-driven file declarations and entrypoint tracking.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Manifest.Assembly.csImplements action-driven assembly definition parsing, keys, and directive application.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Literals.csImplements action-driven parsing for identifiers, dotted names, numeric literals, etc.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Instructions.References.csImplements instruction emission paths that require resolving types/members/tokens.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Declarations.Actions.csImplements top-level declaration dispatch actions.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Debug.csImplements action-driven .line/.language handling and sequence point emission.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Data.csImplements streamed .data emission and reference fixups.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.CustomAttributes.Sequences.csImplements custom-attribute “sequence” value emission.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.CustomAttributes.Actions.csImplements custom-attribute descriptor parsing and blob materialization.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.csAdds per-document lifecycle reset for action-driven compilation.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Conversions.csCentralizes shared state/diagnostics and attribute application helpers.
src/tools/ilasm/src/ILAssembler/Actions/GrammarActions.Bytes.csAdds bytearray parsing that tolerates lexer ambiguities and avoids exceptions.
src/tools/ilasm/src/ilasm/Program.csSwitches output writing to use OutputFile.Write.
src/tools/ilasm/src/ilasm/OutputFile.csAdds retrying output writer to mitigate transient Windows sharing violations.
src/tools/ilasm/README.mdUpdates documentation to describe the action-driven architecture and ref contract pattern.
src/tools/ilasm/ilasm.slnxAdds the ref contract project to the solution.

Comment on lines 14 to +20
<ItemGroup>
<PackageReference Include="Antlr4.Runtime.Standard" Version="$(Antlr4RuntimeStandardVersion)" />
</ItemGroup>

<ItemGroup>
<Compile Remove="ref\**\*.cs" />
</ItemGroup>
Comment on lines +12 to +13
public sealed class Options
{

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This package is internal-only, so API review is not necessary.

CopilotAI review requested due to automatic review settings August 26, 2026 20:30

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 83 out of 88 changed files in this pull request and generated 1 comment.

Comment on lines +397 to +403
[Fact]
public async Task OutputFile_Write_RetriesTransientSharingViolation()
{
if (!OperatingSystem.IsWindows())
{
return;
}
CopilotAI review requested due to automatic review settings August 26, 2026 20:46

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 83 out of 88 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/tools/ilasm/src/ILAssembler/DocumentCompiler.cs:112

  • ParserErrorListener uses TokenSource.SourceName to look up SourceText in loadedDocuments. For synthetic/missing tokens created during error recovery, TokenSource may be PreprocessedTokenSource, whose SourceName can include include-chain decorations (and may not match the raw document keys). Using the underlying InputStream.SourceName aligns with Location.From(...) and should make document lookup robust for recovery tokens.

Comment on lines +8 to +12
public (System.Collections.Immutable.ImmutableArray<Diagnostic>, CompilationResult?) Compile(
System.Collections.Immutable.ImmutableArray<SourceText> documents,
System.Func<string, SourceText> includedDocumentLoader,
System.Func<string, byte[]> resourceLocator,
Options options) { throw null; }
jkoritzinskyand others added 7 commits August 26, 2026 14:47
Use an unbuffered token stream and disable whole-document parse-tree
construction. Retain only bounded declaration and member subtrees while
semantic actions compile them, and stream byte arrays directly.
Split the compiler actions by IL feature to keep the implementation
maintainable and add error-recovery and large-input coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Emit common instruction and method-body forms directly from thin parser
actions without constructing temporary parse subtrees. Keep complex
operands in isolated bounded subtrees for subsequent stack layers.
Preserve instruction diagnostics and deterministic output while reducing
large-method assembly time.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Synthesize floating literals and composed strings directly from parser
rules, and stream switch labels into an action-owned accumulator. Remove
the corresponding bounded instruction subtrees while preserving output
and error recovery.
Also fix odd-length ANSI string padding to match native ilasm.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Remove the generic instruction parse-tree island and emit reference,
token, type, and calli instructions directly after their operand rules.
Each complex operand now retains only its own bounded subtree, preparing
the type and signature rules for independent synthesis.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Convert type, signature, class-name, and member-reference rules to compact
semantic values so reference instructions no longer require parse-tree
subtrees. Preserve internal entity types behind object-valued generated
context slots and materialize them through strongly typed action helpers.
Keep marshalling as the final bounded signature island.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Convert native marshalling, SAFEARRAY variants, IID parameters, and raw
marshal blobs to compact semantic values. Remove the final signature-layer
parse subtree while preserving descriptor bytes and recursive native-type
ordering.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Replace the generic method-declaration and SEH subtrees with direct
method directive actions and synthesized exception-region descriptors.
Keep nested scopes and handlers tree-free while preserving catch type and
label allocation order.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
jkoritzinskyand others added 11 commits August 26, 2026 14:51
Replace class-member and method-header subtrees with synthesized method,
field, property, event, generic, and P/Invoke values. Resolve class-level
method overrides at type close so forward method definitions bind like
native ilasm.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Replace the top-level declaration, namespace-header, and class-header
subtrees with direct dispatch and synthesized type headers. Preserve
class attribute, generic constraint, inheritance, interface, and nested
scope ordering while leaving shared directives in minimal islands.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Convert custom attribute descriptors, named arguments, serialized values,
arrays, object sequences, and field/parameter initializers to semantic
values. Remove their parse-tree islands while preserving pseudo-attribute
lowering and owner binding.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Stream data declaration items and synthesize declarative security, source
mapping, and language directives. Remove their shared parse-tree islands
while preserving label fixups, parent ownership, and PDB state.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Synthesize assembly definitions and references, files, exported types,
resources, vtable fixups, and typedefs. Remove the final parse-tree
islands so BuildParseTree remains disabled for the complete parse.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Synthesize assembly definitions and references, files, exported types,
resources, vtable fixups, and typedefs. Remove the final parse-tree
islands so BuildParseTree remains disabled for the complete parse.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Disable visitor generation and remove the generated visitor interfaces,
parser Accept overrides, explicit forwarding methods, and visitor-only dead
code. The parser action pipeline now owns all semantic traversal.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Replace the remaining GrammarResult wrappers with direct semantic return
values and remove the parse-tree mode stack, which became redundant once
tree construction stayed disabled for the entire parse. Inject the action
object directly through the generated parser.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Add an explicit ILAssembler reference contract so generated parser and
implementation-only types do not define the supported API. Replace
object-typed grammar returns and parser accumulation stacks with concrete
CILParser semantic values and rule-local builders.
Harden every typed return against ANTLR error recovery and add malformed
input mutation coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e2b8e8da-8e3f-4a26-9c8b-73857fabfa6c
Retry transient Windows sharing and lock violations while opening the output file, and cover the IL_Conformance race with a focused regression test.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c986c103-8383-4db4-9b11-9be49dc68a05
@jkoritzinsky
jkoritzinsky changed the base branch from ilasm-pseudoattributes to ilasm-more-testsAugust 26, 2026 22:04

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 83 out of 88 changed files in this pull request and generated 3 comments.

Comment on lines +74 to +75
byte[] resourceData = _resourceLocator(header.Alias);
if (resourceData is null)
Comment on lines +58 to +62
bool neg = text.StartsWith('-');
if (!double.TryParse(text, NumberStyles.Float, CultureInfo.InvariantCulture, out double result))
{
result = neg ? double.MaxValue : double.MinValue;
}
Comment on lines 570 to +573
builder.AddEvent(
evt.Attributes,
builder.GetOrAddString(evt.Name),
evt.Type.Handle);
evt.Type?.Handle ?? default(TypeDefinitionHandle));
}
}

private static bool IsSharingViolation(IOException exception)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do other compilers (e.g. Roslyn, linkers, ...) have a retry logic like this? It does not feel right to compensate for OS race conditions between the file creating/deletion and execution in the compiler.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants

@jkoritzinsky@jkotas