Remove List<T>.Enumerator.MoveNextRare

[stephentoub]

Take 2 on #116150

The JIT work to stack allocate enumerators stops working with List<T> when List<T> gets sufficiently long, e.g. around 1000 elements. At that point, profiling sees the MoveNextRare method used for the last MoveNext as being cold and doesn't inline it. With it not inlined, the boxed enumerator escapes, and the enumerator is then not stack allocated.

(This change only moves the boundary significantly, from ~1000 elements to ~10000 elements. At ~10000, it starts hitting a new limit, due to OSR.)

[Copilot]

Pull Request Overview

This PR optimizes the List<T>.Enumerator implementation by removing the MoveNextRare method to improve JIT stack allocation of enumerators. The change addresses a performance issue where the JIT's stack allocation optimization for enumerators fails when List<T> instances become sufficiently large (around 1000 elements) because the MoveNextRare method is considered cold and not inlined, causing the enumerator to be boxed instead of stack-allocated.

Key changes:

• Inlines the MoveNextRare logic directly into the MoveNext method
• Reorders field declarations and simplifies the enumerator state management
• Changes the end-of-enumeration index marker from _list._size + 1 to -1

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

Related to #118420, cc: @AndyAyersMS

[AndyAyersMS]

FYI there is a similar pattern in PriorityQueue

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

@jkotas, any concerns?

[jkotas]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

Should we go all the way and mark MoveNext() with aggressive inlining?

I do not have a strong opinion either way. This feels like a variant of the code size vs. microbenchmark perf trade off we have faced number of times.

[stephentoub]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

@EgorBo or @AndyAyersMS can comment more authoritatively, but it seems like it's still inlineable even without PGO:
SharpLab

I can mark it with AggressiveInlining, though, if we want to be more sure it happens.

[jkotas]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

[AndyAyersMS]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

In cases where the jit sees the struct enumerator in IL, it applies a fair number of inlining boosts even without PGO:

Invoking compiler for the inlinee method System.Collections.Generic.List`1+Enumerator[int]:MoveNext():bool:this :
...
multiplier in methods of struct increased to 3.
9 ldfld or stfld over arguments which are structs. Multiplier increased to 4.
Inline candidate has arg that feeds range check. Multiplier increased to 5.
Inline candidate is generic and caller is not. Multiplier increased to 7.
Inline candidate has 1 foldable branches. Multiplier increased to 11.
Inline has 1 foldable binary expressions. Multiplier increased to 13.
Inline candidate has an arg that feeds a constant test. Multiplier increased to 14.
Inline candidate callsite is in a loop. Multiplier increased to 17.
calleeNativeSizeEstimate=829
callsiteNativeSizeEstimate=85
benefit multiplier=17
threshold=1445
Native estimate for function size is within threshold for inlining 82.9 <= 144.5 (multiplier = 17)

[stephentoub]

Take 2 on #116150

The JIT work to stack allocate enumerators stops working with List<T> when List<T> gets sufficiently long, e.g. around 1000 elements. At that point, profiling sees the MoveNextRare method used for the last MoveNext as being cold and doesn't inline it. With it not inlined, the boxed enumerator escapes, and the enumerator is then not stack allocated.

(This change only moves the boundary significantly, from ~1000 elements to ~10000 elements. At ~10000, it starts hitting a new limit, due to OSR.)

[Copilot]

Pull Request Overview

This PR optimizes the List<T>.Enumerator implementation by removing the MoveNextRare method to improve JIT stack allocation of enumerators. The change addresses a performance issue where the JIT's stack allocation optimization for enumerators fails when List<T> instances become sufficiently large (around 1000 elements) because the MoveNextRare method is considered cold and not inlined, causing the enumerator to be boxed instead of stack-allocated.

Key changes:

• Inlines the MoveNextRare logic directly into the MoveNext method
• Reorders field declarations and simplifies the enumerator state management
• Changes the end-of-enumeration index marker from _list._size + 1 to -1

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

Related to #118420, cc: @AndyAyersMS

[AndyAyersMS]

FYI there is a similar pattern in PriorityQueue

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

@jkotas, any concerns?

[jkotas]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

Should we go all the way and mark MoveNext() with aggressive inlining?

I do not have a strong opinion either way. This feels like a variant of the code size vs. microbenchmark perf trade off we have faced number of times.

[stephentoub]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

@EgorBo or @AndyAyersMS can comment more authoritatively, but it seems like it's still inlineable even without PGO:
SharpLab

I can mark it with AggressiveInlining, though, if we want to be more sure it happens.

[jkotas]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

[AndyAyersMS]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

In cases where the jit sees the struct enumerator in IL, it applies a fair number of inlining boosts even without PGO:

Invoking compiler for the inlinee method System.Collections.Generic.List`1+Enumerator[int]:MoveNext():bool:this :
...
multiplier in methods of struct increased to 3.
9 ldfld or stfld over arguments which are structs. Multiplier increased to 4.
Inline candidate has arg that feeds range check. Multiplier increased to 5.
Inline candidate is generic and caller is not. Multiplier increased to 7.
Inline candidate has 1 foldable branches. Multiplier increased to 11.
Inline has 1 foldable binary expressions. Multiplier increased to 13.
Inline candidate has an arg that feeds a constant test. Multiplier increased to 14.
Inline candidate callsite is in a loop. Multiplier increased to 17.
calleeNativeSizeEstimate=829
callsiteNativeSizeEstimate=85
benefit multiplier=17
threshold=1445
Native estimate for function size is within threshold for inlining 82.9 <= 144.5 (multiplier = 17)

[stephentoub]

Take 2 on #116150

The JIT work to stack allocate enumerators stops working with List<T> when List<T> gets sufficiently long, e.g. around 1000 elements. At that point, profiling sees the MoveNextRare method used for the last MoveNext as being cold and doesn't inline it. With it not inlined, the boxed enumerator escapes, and the enumerator is then not stack allocated.

(This change only moves the boundary significantly, from ~1000 elements to ~10000 elements. At ~10000, it starts hitting a new limit, due to OSR.)

[Copilot]

Pull Request Overview

This PR optimizes the List<T>.Enumerator implementation by removing the MoveNextRare method to improve JIT stack allocation of enumerators. The change addresses a performance issue where the JIT's stack allocation optimization for enumerators fails when List<T> instances become sufficiently large (around 1000 elements) because the MoveNextRare method is considered cold and not inlined, causing the enumerator to be boxed instead of stack-allocated.

Key changes:

• Inlines the MoveNextRare logic directly into the MoveNext method
• Reorders field declarations and simplifies the enumerator state management
• Changes the end-of-enumeration index marker from _list._size + 1 to -1

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

Related to #118420, cc: @AndyAyersMS

[AndyAyersMS]

FYI there is a similar pattern in PriorityQueue

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

@jkotas, any concerns?

[jkotas]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

Should we go all the way and mark MoveNext() with aggressive inlining?

I do not have a strong opinion either way. This feels like a variant of the code size vs. microbenchmark perf trade off we have faced number of times.

[stephentoub]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

@EgorBo or @AndyAyersMS can comment more authoritatively, but it seems like it's still inlineable even without PGO:
SharpLab

I can mark it with AggressiveInlining, though, if we want to be more sure it happens.

[jkotas]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

[AndyAyersMS]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

In cases where the jit sees the struct enumerator in IL, it applies a fair number of inlining boosts even without PGO:

Invoking compiler for the inlinee method System.Collections.Generic.List`1+Enumerator[int]:MoveNext():bool:this :
...
multiplier in methods of struct increased to 3.
9 ldfld or stfld over arguments which are structs. Multiplier increased to 4.
Inline candidate has arg that feeds range check. Multiplier increased to 5.
Inline candidate is generic and caller is not. Multiplier increased to 7.
Inline candidate has 1 foldable branches. Multiplier increased to 11.
Inline has 1 foldable binary expressions. Multiplier increased to 13.
Inline candidate has an arg that feeds a constant test. Multiplier increased to 14.
Inline candidate callsite is in a loop. Multiplier increased to 17.
calleeNativeSizeEstimate=829
callsiteNativeSizeEstimate=85
benefit multiplier=17
threshold=1445
Native estimate for function size is within threshold for inlining 82.9 <= 144.5 (multiplier = 17)

[stephentoub]

Take 2 on #116150

The JIT work to stack allocate enumerators stops working with List<T> when List<T> gets sufficiently long, e.g. around 1000 elements. At that point, profiling sees the MoveNextRare method used for the last MoveNext as being cold and doesn't inline it. With it not inlined, the boxed enumerator escapes, and the enumerator is then not stack allocated.

(This change only moves the boundary significantly, from ~1000 elements to ~10000 elements. At ~10000, it starts hitting a new limit, due to OSR.)

[Copilot]

Pull Request Overview

This PR optimizes the List<T>.Enumerator implementation by removing the MoveNextRare method to improve JIT stack allocation of enumerators. The change addresses a performance issue where the JIT's stack allocation optimization for enumerators fails when List<T> instances become sufficiently large (around 1000 elements) because the MoveNextRare method is considered cold and not inlined, causing the enumerator to be boxed instead of stack-allocated.

Key changes:

• Inlines the MoveNextRare logic directly into the MoveNext method
• Reorders field declarations and simplifies the enumerator state management
• Changes the end-of-enumeration index marker from _list._size + 1 to -1

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

Related to #118420, cc: @AndyAyersMS

[AndyAyersMS]

FYI there is a similar pattern in PriorityQueue

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

@jkotas, any concerns?

[jkotas]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

Should we go all the way and mark MoveNext() with aggressive inlining?

I do not have a strong opinion either way. This feels like a variant of the code size vs. microbenchmark perf trade off we have faced number of times.

[stephentoub]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

@EgorBo or @AndyAyersMS can comment more authoritatively, but it seems like it's still inlineable even without PGO:
SharpLab

I can mark it with AggressiveInlining, though, if we want to be more sure it happens.

[jkotas]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

[AndyAyersMS]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

In cases where the jit sees the struct enumerator in IL, it applies a fair number of inlining boosts even without PGO:

Invoking compiler for the inlinee method System.Collections.Generic.List`1+Enumerator[int]:MoveNext():bool:this :
...
multiplier in methods of struct increased to 3.
9 ldfld or stfld over arguments which are structs. Multiplier increased to 4.
Inline candidate has arg that feeds range check. Multiplier increased to 5.
Inline candidate is generic and caller is not. Multiplier increased to 7.
Inline candidate has 1 foldable branches. Multiplier increased to 11.
Inline has 1 foldable binary expressions. Multiplier increased to 13.
Inline candidate has an arg that feeds a constant test. Multiplier increased to 14.
Inline candidate callsite is in a loop. Multiplier increased to 17.
calleeNativeSizeEstimate=829
callsiteNativeSizeEstimate=85
benefit multiplier=17
threshold=1445
Native estimate for function size is within threshold for inlining 82.9 <= 144.5 (multiplier = 17)

[stephentoub]

Take 2 on #116150

The JIT work to stack allocate enumerators stops working with List<T> when List<T> gets sufficiently long, e.g. around 1000 elements. At that point, profiling sees the MoveNextRare method used for the last MoveNext as being cold and doesn't inline it. With it not inlined, the boxed enumerator escapes, and the enumerator is then not stack allocated.

(This change only moves the boundary significantly, from ~1000 elements to ~10000 elements. At ~10000, it starts hitting a new limit, due to OSR.)

[Copilot]

Pull Request Overview

This PR optimizes the List<T>.Enumerator implementation by removing the MoveNextRare method to improve JIT stack allocation of enumerators. The change addresses a performance issue where the JIT's stack allocation optimization for enumerators fails when List<T> instances become sufficiently large (around 1000 elements) because the MoveNextRare method is considered cold and not inlined, causing the enumerator to be boxed instead of stack-allocated.

Key changes:

• Inlines the MoveNextRare logic directly into the MoveNext method
• Reorders field declarations and simplifies the enumerator state management
• Changes the end-of-enumeration index marker from _list._size + 1 to -1

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

Related to #118420, cc: @AndyAyersMS

[AndyAyersMS]

FYI there is a similar pattern in PriorityQueue

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

@jkotas, any concerns?

[jkotas]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

Should we go all the way and mark MoveNext() with aggressive inlining?

I do not have a strong opinion either way. This feels like a variant of the code size vs. microbenchmark perf trade off we have faced number of times.

[stephentoub]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

@EgorBo or @AndyAyersMS can comment more authoritatively, but it seems like it's still inlineable even without PGO:
SharpLab

I can mark it with AggressiveInlining, though, if we want to be more sure it happens.

[jkotas]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

[AndyAyersMS]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

In cases where the jit sees the struct enumerator in IL, it applies a fair number of inlining boosts even without PGO:

Invoking compiler for the inlinee method System.Collections.Generic.List`1+Enumerator[int]:MoveNext():bool:this :
...
multiplier in methods of struct increased to 3.
9 ldfld or stfld over arguments which are structs. Multiplier increased to 4.
Inline candidate has arg that feeds range check. Multiplier increased to 5.
Inline candidate is generic and caller is not. Multiplier increased to 7.
Inline candidate has 1 foldable branches. Multiplier increased to 11.
Inline has 1 foldable binary expressions. Multiplier increased to 13.
Inline candidate has an arg that feeds a constant test. Multiplier increased to 14.
Inline candidate callsite is in a loop. Multiplier increased to 17.
calleeNativeSizeEstimate=829
callsiteNativeSizeEstimate=85
benefit multiplier=17
threshold=1445
Native estimate for function size is within threshold for inlining 82.9 <= 144.5 (multiplier = 17)

[stephentoub]

Take 2 on #116150

The JIT work to stack allocate enumerators stops working with List<T> when List<T> gets sufficiently long, e.g. around 1000 elements. At that point, profiling sees the MoveNextRare method used for the last MoveNext as being cold and doesn't inline it. With it not inlined, the boxed enumerator escapes, and the enumerator is then not stack allocated.

(This change only moves the boundary significantly, from ~1000 elements to ~10000 elements. At ~10000, it starts hitting a new limit, due to OSR.)

[Copilot]

Pull Request Overview

This PR optimizes the List<T>.Enumerator implementation by removing the MoveNextRare method to improve JIT stack allocation of enumerators. The change addresses a performance issue where the JIT's stack allocation optimization for enumerators fails when List<T> instances become sufficiently large (around 1000 elements) because the MoveNextRare method is considered cold and not inlined, causing the enumerator to be boxed instead of stack-allocated.

Key changes:

• Inlines the MoveNextRare logic directly into the MoveNext method
• Reorders field declarations and simplifies the enumerator state management
• Changes the end-of-enumeration index marker from _list._size + 1 to -1

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

Related to #118420, cc: @AndyAyersMS

[AndyAyersMS]

FYI there is a similar pattern in PriorityQueue

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

@jkotas, any concerns?

[jkotas]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

Should we go all the way and mark MoveNext() with aggressive inlining?

I do not have a strong opinion either way. This feels like a variant of the code size vs. microbenchmark perf trade off we have faced number of times.

[stephentoub]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

@EgorBo or @AndyAyersMS can comment more authoritatively, but it seems like it's still inlineable even without PGO:
SharpLab

I can mark it with AggressiveInlining, though, if we want to be more sure it happens.

[jkotas]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

[AndyAyersMS]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

In cases where the jit sees the struct enumerator in IL, it applies a fair number of inlining boosts even without PGO:

Invoking compiler for the inlinee method System.Collections.Generic.List`1+Enumerator[int]:MoveNext():bool:this :
...
multiplier in methods of struct increased to 3.
9 ldfld or stfld over arguments which are structs. Multiplier increased to 4.
Inline candidate has arg that feeds range check. Multiplier increased to 5.
Inline candidate is generic and caller is not. Multiplier increased to 7.
Inline candidate has 1 foldable branches. Multiplier increased to 11.
Inline has 1 foldable binary expressions. Multiplier increased to 13.
Inline candidate has an arg that feeds a constant test. Multiplier increased to 14.
Inline candidate callsite is in a loop. Multiplier increased to 17.
calleeNativeSizeEstimate=829
callsiteNativeSizeEstimate=85
benefit multiplier=17
threshold=1445
Native estimate for function size is within threshold for inlining 82.9 <= 144.5 (multiplier = 17)

[stephentoub]

Take 2 on #116150

The JIT work to stack allocate enumerators stops working with List<T> when List<T> gets sufficiently long, e.g. around 1000 elements. At that point, profiling sees the MoveNextRare method used for the last MoveNext as being cold and doesn't inline it. With it not inlined, the boxed enumerator escapes, and the enumerator is then not stack allocated.

(This change only moves the boundary significantly, from ~1000 elements to ~10000 elements. At ~10000, it starts hitting a new limit, due to OSR.)

[Copilot]

Pull Request Overview

This PR optimizes the List<T>.Enumerator implementation by removing the MoveNextRare method to improve JIT stack allocation of enumerators. The change addresses a performance issue where the JIT's stack allocation optimization for enumerators fails when List<T> instances become sufficiently large (around 1000 elements) because the MoveNextRare method is considered cold and not inlined, causing the enumerator to be boxed instead of stack-allocated.

Key changes:

• Inlines the MoveNextRare logic directly into the MoveNext method
• Reorders field declarations and simplifies the enumerator state management
• Changes the end-of-enumeration index marker from _list._size + 1 to -1

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

Related to #118420, cc: @AndyAyersMS

[AndyAyersMS]

FYI there is a similar pattern in PriorityQueue

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

@jkotas, any concerns?

[jkotas]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

Should we go all the way and mark MoveNext() with aggressive inlining?

I do not have a strong opinion either way. This feels like a variant of the code size vs. microbenchmark perf trade off we have faced number of times.

[stephentoub]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

@EgorBo or @AndyAyersMS can comment more authoritatively, but it seems like it's still inlineable even without PGO:
SharpLab

I can mark it with AggressiveInlining, though, if we want to be more sure it happens.

[jkotas]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

[AndyAyersMS]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

In cases where the jit sees the struct enumerator in IL, it applies a fair number of inlining boosts even without PGO:

Invoking compiler for the inlinee method System.Collections.Generic.List`1+Enumerator[int]:MoveNext():bool:this :
...
multiplier in methods of struct increased to 3.
9 ldfld or stfld over arguments which are structs. Multiplier increased to 4.
Inline candidate has arg that feeds range check. Multiplier increased to 5.
Inline candidate is generic and caller is not. Multiplier increased to 7.
Inline candidate has 1 foldable branches. Multiplier increased to 11.
Inline has 1 foldable binary expressions. Multiplier increased to 13.
Inline candidate has an arg that feeds a constant test. Multiplier increased to 14.
Inline candidate callsite is in a loop. Multiplier increased to 17.
calleeNativeSizeEstimate=829
callsiteNativeSizeEstimate=85
benefit multiplier=17
threshold=1445
Native estimate for function size is within threshold for inlining 82.9 <= 144.5 (multiplier = 17)

[stephentoub]

Take 2 on #116150

The JIT work to stack allocate enumerators stops working with List<T> when List<T> gets sufficiently long, e.g. around 1000 elements. At that point, profiling sees the MoveNextRare method used for the last MoveNext as being cold and doesn't inline it. With it not inlined, the boxed enumerator escapes, and the enumerator is then not stack allocated.

(This change only moves the boundary significantly, from ~1000 elements to ~10000 elements. At ~10000, it starts hitting a new limit, due to OSR.)

[Copilot]

Pull Request Overview

This PR optimizes the List<T>.Enumerator implementation by removing the MoveNextRare method to improve JIT stack allocation of enumerators. The change addresses a performance issue where the JIT's stack allocation optimization for enumerators fails when List<T> instances become sufficiently large (around 1000 elements) because the MoveNextRare method is considered cold and not inlined, causing the enumerator to be boxed instead of stack-allocated.

Key changes:

• Inlines the MoveNextRare logic directly into the MoveNext method
• Reorders field declarations and simplifies the enumerator state management
• Changes the end-of-enumeration index marker from _list._size + 1 to -1

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

Related to #118420, cc: @AndyAyersMS

[AndyAyersMS]

FYI there is a similar pattern in PriorityQueue

[stephentoub]

@EgorBot -arm -amd -intel

usingBenchmarkDotNet.Attributes;usingBenchmarkDotNet.Running;BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);[MemoryDiagnoser(false)]publicclassBench{[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumList(List<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}[Benchmark][ArgumentsSource(nameof(GetLists))]publicintSumEnumerable(IEnumerable<int>list){intsum=0;foreach(intiteminlist){sum+=item;}returnsum;}publicstaticIEnumerable<List<int>>GetLists()=>fromcountinnewint[]{1,10,1_000}selectEnumerable.Range(0,count).ToList();}

[stephentoub]

@jkotas, any concerns?

[jkotas]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

Should we go all the way and mark MoveNext() with aggressive inlining?

I do not have a strong opinion either way. This feels like a variant of the code size vs. microbenchmark perf trade off we have faced number of times.

[stephentoub]

I assume that this will regress perf for common cases without PGO (NAOT and probably Mono too) since MoveNext() hot path is not going to be inlined anymore. Is that correct?

@EgorBo or @AndyAyersMS can comment more authoritatively, but it seems like it's still inlineable even without PGO:
SharpLab

I can mark it with AggressiveInlining, though, if we want to be more sure it happens.

[jkotas]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

[AndyAyersMS]

it seems like it's still inlineable even without PGO:

Cool, LGTM then.

In cases where the jit sees the struct enumerator in IL, it applies a fair number of inlining boosts even without PGO:

Invoking compiler for the inlinee method System.Collections.Generic.List`1+Enumerator[int]:MoveNext():bool:this :
...
multiplier in methods of struct increased to 3.
9 ldfld or stfld over arguments which are structs. Multiplier increased to 4.
Inline candidate has arg that feeds range check. Multiplier increased to 5.
Inline candidate is generic and caller is not. Multiplier increased to 7.
Inline candidate has 1 foldable branches. Multiplier increased to 11.
Inline has 1 foldable binary expressions. Multiplier increased to 13.
Inline candidate has an arg that feeds a constant test. Multiplier increased to 14.
Inline candidate callsite is in a loop. Multiplier increased to 17.
calleeNativeSizeEstimate=829
callsiteNativeSizeEstimate=85
benefit multiplier=17
threshold=1445
Native estimate for function size is within threshold for inlining 82.9 <= 144.5 (multiplier = 17)