JIT: Slow spills to stack in a loop on Linux vs Windows

[EgorBo]

The following program (reported by @WhiteBlackGoose):

public class Program
{
 struct RangeEnumerator
 {
 private readonly int to;
 private readonly int step;
 private int curr;

 public bool MoveNext()
 {
 curr += step;
 return curr != to;
 }

 public int Current => curr;
 }

 static int EnumerateItAll(RangeEnumerator enumerator)
 {
 var a = 0;
 while (enumerator.MoveNext())
 a += enumerator.Current;
 return a;
 }
}

here is the assembly diff for EnumerateItAll method between Win-x64 and Linux-x64: https://www.diffchecker.com/NorefcMS
Windows one looks much better and there are no spills in the Loop body leading to much better performance.
I've not investigated this one yet - from my understanding RangeEnumerator is passed by stack on win-x64 ABI and via registers on lin-x64. Is there anything we can do here for Linux or it's a dup for some "first class structs promotion" issue? cc @dotnet/jit-contrib @kunalspathak

category:cq
theme:stack-allocation

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

---

The following program (reported by @WhiteBlackGoose):

public class Program
{
 struct RangeEnumerator
 {
 private readonly int to;
 private readonly int step;
 private int curr;

 public bool MoveNext()
 {
 curr += step;
 return curr != to;
 }

 public int Current => curr;
 }

 static int EnumerateItAll(RangeEnumerator enumerator)
 {
 var a = 0;
 while (enumerator.MoveNext())
 a += enumerator.Current;
 return a;
 }
}

here is the assembly diff for EnumerateItAll method between Win-x64 and Linux-x64: https://www.diffchecker.com/NorefcMS
Windows one looks much better and there is no spills in the Loop body leading to much better performance.
I've not investigated this one yet - from my understanding RangeEnumerator is passed by stack on win-x64 ABI and via registers on lin-x64. Is there anything we can do here for Linux ? cc @dotnet/jit-contrib @kunalspathak

Author:	EgorBo
Assignees:	-
Labels:	tenet-performance, area-CodeGen-coreclr, untriaged
Milestone:	-

[WhiteBlackGoose]

To add a bit more details:
Here's the original method:

public int ForeachWithRangeEnumeratorRawWithLocal()
{
 var enumerator = (0..(Iterations - 1)).GetEnumerator(); 
 return EnumerateItAll(enumerator);

 static int EnumerateItAll(RangeEnumerator enumerator)
 {
 var a = 0;
 while (enumerator.MoveNext())
 a += enumerator.Current;
 return a;
 }
}

At the point where return happens, it emits a call on Windows, but a jmp on Linux.

Here's the full codegen for both linux and windows: gist.

Sorry if those details are irrelevant 😅, hope they add something to the discussion.

[AndyAyersMS]

Seems like this runs into a limitation of #36862 where if the number of fields of a multi-reg struct arg doesn't match the number of registers then the struct is not promoted. Here we have 3 int fields taking up 12 bytes, passed in 2 8-byte registers.

Would be easier to verify if the struct promotion code left a better debug trail.

[AndyAyersMS]

Actually we hit an earlier bail out; we only support promotion on x64 if there are 1 or 2 fields:

runtime/src/coreclr/jit/lclvars.cpp

Lines 1958 to 1962 in 83f6621

	unsigned fieldCnt = structPromotionInfo.fieldCnt;
	if (fieldCnt > MAX_MULTIREG_COUNT)
	{
	canPromote = false;
	}

[jakobbotsch]

Codegen with physical promotion enabled:

; Assembly listing for method Program:EnumerateItAll(Program+RangeEnumerator):int
; Emitting BLENDED_CODE for X64 with AVX512 - Unix
; optimized code
; rbp based frame
; fully interruptible
; No PGO data
; 0 inlinees with PGO data; 2 single block inlinees; 0 inlinees without PGO data
; invoked as altjit
; Final local variable assignments
;
; V00 arg0 [V00,T02] ( 5, 5 ) struct (16) [rbp-10H] do-not-enreg[SFA] multireg-arg ld-addr-op single-def
; V01 loc0 [V01,T01] ( 4, 10 ) int -> rdi
;# V02 OutArgs [V02 ] ( 1, 1 ) struct ( 0) [rsp+00H] do-not-enreg[XS] addr-exposed "OutgoingArgSpace"
; V03 tmp1 [V03,T00] ( 8, 20 ) int -> rax "V00.[008..012)"
; V04 cse0 [V04,T03] ( 3, 6 ) int -> rsi "CSE - aggressive"
; V05 cse1 [V05,T04] ( 3, 6 ) int -> rdx "CSE - aggressive"
;
; Lcl frame size = 16

G_M19832_IG01: ;; offset=0000H
 push rbp
 sub rsp, 16
 lea rbp, [rsp+10H]
 mov qword ptr [rbp-10H], rdi
 mov dword ptr [rbp-08H], esi
 ;; size=17 bbWeight=1 PerfScore 3.75
G_M19832_IG02: ;; offset=0011H
 mov eax, dword ptr [rbp-08H]
 xor edi, edi
 mov esi, dword ptr [rbp-0CH]
 add eax, esi
 mov edx, dword ptr [rbp-10H]
 cmp eax, edx
 je SHORT G_M19832_IG04
 align [0 bytes for IG03]
 ;; size=17 bbWeight=1 PerfScore 4.75
G_M19832_IG03: ;; offset=0022H
 add edi, eax
 add eax, esi
 cmp eax, edx
 jne SHORT G_M19832_IG03
 ;; size=8 bbWeight=4 PerfScore 7.00
G_M19832_IG04: ;; offset=002AH
 mov eax, edi
 ;; size=2 bbWeight=1 PerfScore 0.25
G_M19832_IG05: ;; offset=002CH
 add rsp, 16
 pop rbp
 ret
 ;; size=6 bbWeight=1 PerfScore 1.75

; Total bytes of code 50, prolog size 17, PerfScore 22.50, instruction count 21, allocated bytes for code 50 (MethodHash=9755b287) for method Program:EnumerateItAll(Program+RangeEnumerator):int
; ============================================================

We still spill as part of homing, but most importantly the loop ends up optimized.