Managed ilasm: comprehensive parity fixes with native ilasm

[jkoritzinsky]

Managed ilasm parity fixes

This PR brings the managed ilasm (src/tools/ilasm) to near-parity with the native ilasm across 2,868 ilproj test files in the repo. Starting from ~2,128 managed ilasm failures, this work reduces real (non-cosmetic) differences to zero actionable items.

Summary of changes (53 commits, 23 files, +12,035 / -6,617 lines)

Parser & Grammar Fixes

• Fixed ANTLR grammar for .language, .line (QSTRING support), value/instance keywords in identifiers, multiple ddItem without braces, &label references, atOpt with integer, empty pinvokeimpl(), security attribute blob type ordering, function pointer syntax
• Added ANTLR parser error listener to DocumentCompiler for strict diagnostics
• Fixed preprocessor #define macro expansion to re-lex multi-token values
• Regenerated ANTLR parser after all grammar changes

Metadata Emission Fixes

• MemberRef → MethodDef/FieldDef resolution: Resolve local method/field references to definition tokens (matching native ilasm behavior)
• TypeRef → TypeDef resolution: Lazy TypeRef tracking with PseudoHandle for signature encoding. TypeRefs whose resolution scope matches the current assembly resolve to local TypeDef handles. Signature rewriter remaps PseudoHandle-based coded indices in all signature blobs and IL instruction tokens are backpatched.
• Field attribute flags: HasDefault, HasFieldMarshal, HasFieldRVA, PinvokeImpl
• Method attributes: Auto-instance calling convention, auto-RTSpecialName|SpecialName for .ctor/.cctor, PinvokeImpl flag
• Param emission: Always emit Param rows for explicit parameters, auto-generate A_N names for unnamed parameters
• ClassLayout: Emit for explicit layout types even without .pack/.size
• GenericParamConstraint: Sort by Owner handle during emission
• Custom attributes: Fix type-level and top-level handlers, blob prolog (WriteUInt16 not WriteInt32), module vs assembly ownership
• Stackreserve: Pass directive value to PEHeaderBuilder
• Locals: Build LOCAL_SIG standalone signature from parsed .locals declarations
• Corelib TypeRef redirect: Normalize different corelib assembly names
• Primitive type codes: Emit correct codes for well-known corelib types (System.String → String, etc.)
• Leading-dot type names: Position 0 dot is part of the name, not a namespace separator
• DebuggableAttribute: Deferred to BuildImage() so assembly refs are available
• Vararg signatures: Fix parameter count (exclude sentinels) and parent resolution

Signature Rewriter (new)

• Rewrites all signature blobs (field, method, standalone, property, TypeSpec, MethodSpec) from PseudoHandle-based TypeRef coded indices to resolved real handles
• Fixed GetModifiedType to write modifier as raw coded index (not full type encoding)
• Fixed GetArrayType to emit ELEMENT_TYPE_ARRAY prefix byte

IL Body Fixes

• TypeRef IL token backpatching for instr_type, instr_tok, and instr_field mdtoken paths
• Workaround for BlobBuilder chunk boundary corruption (InstructionEncoder produces corrupted IL when mixing OpCode with direct CodeBuilder writes across BlobBuilder chunk boundaries #127261) — increased InstructionEncoderBlobBuilder initial capacity to 4096
• Array shape encoding fix for multi-dimensional arrays (contiguous sizes/lower bounds only)
• Security attribute blob SQSTRING quote stripping

Test Coverage

• 236 unit tests (up from ~25 at start), covering all major fix categories
• Tests use CompileAndGetReader to verify metadata byte-level correctness

Comparison results (2,868 ilproj files)

Metric	Before	After
Matching (byte-identical ildasm)	0	9
Module-name-only diffs	—	1,862
Cascading-only diffs	—	676
Real primary diffs	2,128+ failures	420 (all cosmetic)
Managed ilasm failures	2,128	4 (2 TLS, 2 65K+ generics)

The remaining 420 files with "real" diffs are all non-actionable: custom attribute metadata ordering (246), PE header line ordering (44), field RVA section placement (21), assembly metadata cosmetic (19), corelib ref name cascading (12), ildasm typedef presentation (4), and override TypeRef formatting (3).

Note

This PR was authored with the assistance of GitHub Copilot.

[azure-pipelines]

Azure Pipelines:
Successfully started running 5 pipeline(s).
11 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR substantially expands managed ilasm parser/emitter parity with native ilasm, including multi-document compilation, grammar fixes, metadata/token resolution adjustments, and a large suite of new unit tests validating metadata/IL byte correctness.

Changes:

• Extend the compiler pipeline (preprocessor + parser) to support multi-document input and stricter parser diagnostics.
• Improve preprocessor macro expansion by re-lexing macro values into correct token streams.
• Add broad managed ilasm regression/unit test coverage across directives, signatures, metadata emission, and IL encoding.

Reviewed changes

Copilot reviewed 47 out of 51 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs	New tests for .vtfixup emission and .sdata layout validation.
src/tools/ilasm/tests/ILAssembler.Tests/TypeSignatureTests.cs	New test for multidimensional array bounds parsing.
src/tools/ilasm/tests/ILAssembler.Tests/TypeReferenceTests.cs	New tests for .this/.base/.nester diagnostics and TypeRef→TypeDef resolution/backpatching behavior.
src/tools/ilasm/tests/ILAssembler.Tests/TypedefTests.cs	New tests for typedef alias resolution and missing-alias diagnostics.
src/tools/ilasm/tests/ILAssembler.Tests/SyntaxTests.cs	New tests for string escape handling, numeric literal diagnostics, and parser error reporting.
src/tools/ilasm/tests/ILAssembler.Tests/SourceDirectiveTests.cs	New tests for .language/.line directives and embedded PDB generation behavior.
src/tools/ilasm/tests/ILAssembler.Tests/SecurityTests.cs	New tests for security directive diagnostics and DeclSecurity emission.
src/tools/ilasm/tests/ILAssembler.Tests/PropertyTests.cs	New tests for property emission, initOpt constants, and property-owned custom attributes.
src/tools/ilasm/tests/ILAssembler.Tests/PreprocessedTokenSourceTests.cs	Update preprocessor construction and add macro re-lexing tests.
src/tools/ilasm/tests/ILAssembler.Tests/ParameterTests.cs	New tests for param constants, param row emission, signature rewrite correctness, and ldarg-by-name regression.
src/tools/ilasm/tests/ILAssembler.Tests/ModuleTests.cs	New tests for module naming defaults and module-level field handling.
src/tools/ilasm/tests/ILAssembler.Tests/MethodTests.cs	New tests for vararg, overrides/MethodImpl, calling convention inference, and modopt preservation.
src/tools/ilasm/tests/ILAssembler.Tests/MemberReferenceTests.cs	Placeholder test file added for future coverage.
src/tools/ilasm/tests/ILAssembler.Tests/LocalTests.cs	New tests for locals/name lookup and MemberRef→Def resolution behavior.
src/tools/ilasm/tests/ILAssembler.Tests/InteropTests.cs	New tests for pinvoke parsing/diagnostics and PinvokeImpl emission.
src/tools/ilasm/tests/ILAssembler.Tests/InstructionTests.cs	New tests for label fixups, prefix/opcode parsing, and FieldRVA emission.
src/tools/ilasm/tests/ILAssembler.Tests/GenericTests.cs	New tests for generic parameter diagnostics and constraint emission correctness.
src/tools/ilasm/tests/ILAssembler.Tests/FunctionPointerTests.cs	New tests for function pointer signature encoding.
src/tools/ilasm/tests/ILAssembler.Tests/ExportedTypeTests.cs	New tests for exported type diagnostics, forwarding, and .export behavior.
src/tools/ilasm/tests/ILAssembler.Tests/ExceptionHandlingTests.cs	New tests for EH blocks and label/offset handling.
src/tools/ilasm/tests/ILAssembler.Tests/EventTests.cs	New test ensuring event-owned custom attributes are emitted.
src/tools/ilasm/tests/ILAssembler.Tests/DocumentCompilerTestHelpers.cs	New helper utilities for compilation, diagnostics, and token operand decoding.
src/tools/ilasm/tests/ILAssembler.Tests/DataTests.cs	New test for invalid metadata token diagnostics.
src/tools/ilasm/tests/ILAssembler.Tests/CustomAttributeTests.cs	New tests for custom attribute blob parsing and emission correctness.
src/tools/ilasm/tests/ILAssembler.Tests/ILAssembler.Tests.csproj	Link in shared ILOpcode metadata for tests that decode IL bodies.
src/tools/ilasm/src/ILAssembler/PreprocessedTokenSource.cs	Add macro value re-lexing, exposed defined-variable state, and constructor updates.
src/tools/ilasm/src/ILAssembler/Options.cs	Add OutputFileName option to drive default module naming.
src/tools/ilasm/src/ILAssembler/NameHelpers.cs	Treat leading-dot type names as name (not namespace separator).
src/tools/ilasm/src/ILAssembler/NamedElementList.cs	Allow duplicate names with first-wins lookup behavior.
src/tools/ilasm/src/ILAssembler/gen/ilasm-generator.csproj	Improve ANTLR generated path rewrite robustness.
src/tools/ilasm/src/ILAssembler/gen/CILVisitor.cs	Regenerated visitor surface to match grammar updates.
src/tools/ilasm/src/ILAssembler/gen/CILLexer.tokens	Regenerated token map after grammar changes.
src/tools/ilasm/src/ILAssembler/gen/CILBaseVisitor.cs	Regenerated base visitor after grammar changes.
src/tools/ilasm/src/ILAssembler/gen/CIL.g4	Major grammar adjustments (directives, tokens, keywords, vararg, byte blobs, etc.).
src/tools/ilasm/src/ILAssembler/DocumentCompiler.cs	Add multi-document compile overload; add strict parser error listener; persist preprocessor defines across documents.
src/tools/ilasm/src/ILAssembler/Diagnostic.cs	Add diagnostic ID/template for excessive generic parameter counts.
src/tools/ilasm/src/ILAssembler/BlobBuilderExtensions.cs	Emit pseudo-handle coded indices for TypeRef entities in signatures.
src/tools/ilasm/src/ilasm/Program.cs	Read/compile multiple input files; normalize native-style args; plumb OutputFileName into options.
src/tools/ilasm/src/ilasm/IlasmRootCommand.cs	Add native-style option aliases (e.g., -OUTPUT, -DLL, etc.).
src/tools/ilasm/KNOWN-ISSUES.md	Document current known limitation (TLS RVA statics).

Comments suppressed due to low confidence (3)

src/tools/ilasm/src/ILAssembler/NamedElementList.cs:85

• NamedElementList now allows duplicate names (first-wins) via TryAdd, but Remove/RemoveAt still unconditionally remove the name mapping. If a non-mapped duplicate is removed, this drops the lookup entry for the still-existing first element; if the mapped element is removed, name lookup should fall back to the next remaining element with the same name. This can break name-based lookups after removals.

 public bool Remove(T item)
{
bool result = _elements.Remove(item);
if (result)
{
_elementsByName.Remove(item.Name);
}
return result;
}
public void RemoveAt(int index)
{
T element = _elements[index];
_elements.RemoveAt(index);
_elementsByName.Remove(element.Name);
}

src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs:119

• Same endianness issue as above: these values are read from PE bytes (little-endian). Use BinaryPrimitives.Read*LittleEndian rather than BitConverter so the test behaves correctly on big-endian targets.
  src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs:168
• Same endianness issue: slot count/RVA/tokens are read from PE bytes (little-endian). Using BitConverter makes the test endian-dependent; prefer BinaryPrimitives.Read*LittleEndian here too.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[Copilot]

Pull request overview

Copilot reviewed 47 out of 51 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/tools/ilasm/tests/ILAssembler.Tests/VTableTests.cs:64

• These reads interpret PE/CLR structures which are defined as little-endian. Using BitConverter makes the test endianness-dependent. Please use BinaryPrimitives.Read*LittleEndian here (and similarly for the other BitConverter reads later in this file).
  src/tools/ilasm/tests/ILAssembler.Tests/InstructionTests.cs:88
• This test is reading a little-endian PE value. BitConverter makes the test endianness-dependent; use BinaryPrimitives.ReadInt32LittleEndian instead (and update the other BitConverter reads in this file as well).
  src/tools/ilasm/src/ILAssembler/Options.cs:139
• The XML comment says this is "filename only, no directory", but the property doesn't enforce that. Either normalize to Path.GetFileName in the setter / consumer, or adjust the comment to avoid stating an invariant that isn't guaranteed.

 /// <summary>
/// Output file name (filename only, no directory). Used as default module name when no .module directive is present.
/// </summary>
public string? OutputFileName { get; set; }

[Copilot]

Pull request overview

Copilot reviewed 47 out of 51 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/tools/ilasm/src/ILAssembler/Options.cs:139

• Options.OutputFileName is new public API surface. With no linked api-approved issue in the PR metadata, this needs either API approval linkage or reducing the visibility (and adjusting tests accordingly) if it’s meant to be internal implementation detail.

 /// <summary>
/// Output file name (filename only, no directory). Used as default module name when no .module directive is present.
/// </summary>
public string? OutputFileName { get; set; }

src/tools/ilasm/src/ILAssembler/NamedElementList.cs:48

• Add/Insert now use TryAdd so the first duplicate name wins, but the rest of the type still assumes a 1:1 mapping between name and element (e.g., indexer setter overwrites, and Remove/RemoveAt always remove the name key). With duplicates present, the name map can become stale or lose the winning entry. Consider making the name-index semantics consistent across mutation operations (e.g., only remove the key when it points to the removed element, and re-point it to the next remaining element with the same name).

 public void Add(T item)
{
_elements.Add(item);
// Use TryAdd to keep the first element for name lookup when duplicate names exist.
// This matches native ilasm behavior where duplicate generic parameter names are allowed
// and the first definition wins for name-based lookup.
_elementsByName.TryAdd(item.Name, item);
}

[jkoritzinsky]

OK stack is now updated. Once we get #132346 in, I'll start on those remaining issues.