Unroll SequenceEqual(ref byte, ref byte, nuint) in JIT

[EgorBo]

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.SequenceEqual(data);boolTest2(Span<byte>data)=>data.StartsWith("test"u8);

Codegen diff: https://www.diffchecker.com/E1laymuB/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side.
Works for StartsWith.

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}publicstaticclassPaths{publicstaticReadOnlySpan<byte>Json=>"/json"u8;publicstaticReadOnlySpan<byte>Plaintext=>"/plaintext"u8;}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

(the difference should be bigger when SIMD is enabled)

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest(ReadOnlySpan<byte>data){return"hello world!"u8.SequenceEqual(data);}

Codegen diff: https://www.diffchecker.com/0VOpmvMj/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side (which is likely the case for e.g. StartsWith).

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

Author:	EgorBo
Assignees:	EgorBo
Labels:	area-CodeGen-coreclr
Milestone:	-

[gfoidl]

Can be dasm for Test1 be something like

G_M000_IG01: ;; offset=0000HG_M000_IG02: ;; offset=0000H 488B01 movrax, bword ptr [rcx] 8B5108 movedx, dword ptr [rcx+08H] 83FA0B cmpedx,117404je SHORT G_M000_IG04G_M000_IG03: ;; offset=000BH 33C0 xoreax,eax EB24 jmp SHORT G_M000_IG05G_M000_IG04: ;; offset=000FH 48BA68656C6C6F20776F movrdx,0x6F77206F6C6C6568483310xorrdx, qword ptr [rax] 48B96C6F20776F726C64 movrcx,0x646C726F77206F6C48334803xorrcx, qword ptr [rax+03H] 480BD1 orrdx,rcx 0F94C0 sete al 0FB6C0 movzxrax,alG_M000_IG05: ;; offset=0033H C3 ret; Total bytes of code 52

?
So that the "constant" LHS (here "hello world"u8) is read as long-constant instead of loaded from memory. Save two memory loads.

The assembly above is produced by this simple C# approach.

Code

usingSystem.Runtime.CompilerServices;usingSystem.Runtime.InteropServices;ReadOnlySpan<byte>test="hello world"u8;Console.WriteLine(Test1(test));
#if !DEBUGfor(inti=0;i<100;++i){if(i%10==0)Thread.Sleep(100);_=Test1(test);}
#endif
staticboolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.FastSequenceEqual(data);internalstaticclassMySpanExtensions{[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolFastSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr0=Unsafe.ReadUnaligned<long>(refrightRef);longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}thrownewNotSupportedException();}}

PS: the XOR-trick here is 👍🏻

[jakobbotsch]

Are you sure this is better than the naive version for ARM64 with CCMPs? What is the ARM64 codegen diff if you create AND(EQ(IND, IND), EQ(IND, IND)) instead?

[EgorBo]

The current codegen is (comparing 16 bytes):

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]CA030021 eor x1, x1, x3F9400000 ldr x0,[x0]F9400042 ldr x2,[x2]CA020000 eor x0, x0, x2AA000020 orr x0, x1, x0F100001F cmp x0, #09A9F17E0 cset x0, eq

cmp version presumably needs ifConversion path? Because here is what I see when I follow your suggestion:

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]EB03003F cmp x1, x39A9F17E1 cset x1, eqF9400000 ldr x0,[x0]F9400042 ldr x2,[x2]EB02001F cmp x0, x29A9F17E0 cset x0, eqEA00003F tst x1, x09A9F07E0 cset x0, ne

so we need to either do this opt in codegen or earlier for that. For me arm64 codegen doesn't look too bad, it's still better than not unrolled.

[jakobbotsch]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

[EgorBo]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

still doesn't want to convert to CCMP, IsInvariantInRange check fails, presumably because of IND side effects. Still, I think the current version is better than non-unrolled

[jakobbotsch]

You should be able to insert it in the right order so that there is no interference, e.g. probably

t0 = IND
t1 = IND
t2 = IND
t3 = IND
t4 = EQ(t0, t1)
t5 = EQ(t2, t3)
t6 = AND(t4, t5)

[jakobbotsch]

Although it's a bit odd there would be interference even with

t0 =IND
t1 =IND
t2 =EQ(t0, t1)
t3 =IND
t4 =IND
t5 =EQ(t3, t4)
t6 =AND(t2, t5)

Probably something I should take a look at.

[EgorBo]

but just in case I pushed a change to move all IND nodes to the front

[gfoidl]

I have a general question: why needs this to be done in JIT and not in managed code? Is it about throughput and / or IL-size?
In some areas there's native -> managed, here it's the opposite.
This is not a rant, just a question out of curiosity. And it makes contributing a bit harder (at least for me, as I don't know the internals of JIT very much).

To implement this in pure C# something like RuntimeHelpers.IsKnownConstant(ReadOnlySpan<byte>) is missing.
Assuming such a method exists, then it could look like:

Example managed implementation

[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(/* missing piece */RuntimeHelpers.IsKnownConstant(left)){if(len>=sizeof(int)&&len<=2*sizeof(int)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);intl0=Unsafe.ReadUnaligned<int>(refleftRef);intr0=Unsafe.ReadUnaligned<int>(refrightRef);intl1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refleftRef,len-sizeof(int)));intr1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refrightRef,len-sizeof(int)));intt0=l0^r0;intt1=l1^r1;intt=t0|t1;returnt==0;}if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longr0=Unsafe.ReadUnaligned<long>(refrightRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}if(Vector128.IsHardwareAccelerated&&len>=(uint)Vector128<byte>.Count&&len<=2*(uint)Vector128<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector128<byte>l0=Vector128.LoadUnsafe(refleftRef);Vector128<byte>r0=Vector128.LoadUnsafe(refrightRef);Vector128<byte>t0=l0^r0;Vector128<byte>l1=Vector128.LoadUnsafe(refleftRef,len-(uint)Vector128<byte>.Count);Vector128<byte>r1=Vector128.LoadUnsafe(refrightRef,len-(uint)Vector128<byte>.Count);Vector128<byte>t1=l1^r1;Vector128<byte>t=t0|t1;returnt==Vector128<byte>.Zero;}if(Vector256.IsHardwareAccelerated&&len>=(uint)Vector256<byte>.Count&&len<=2*(uint)Vector256<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector256<byte>l0=Vector256.LoadUnsafe(refleftRef);Vector256<byte>r0=Vector256.LoadUnsafe(refrightRef);Vector256<byte>t0=l0^r0;Vector256<byte>l1=Vector256.LoadUnsafe(refleftRef,len-(uint)Vector256<byte>.Count);Vector256<byte>r1=Vector256.LoadUnsafe(refrightRef,len-(uint)Vector256<byte>.Count);Vector256<byte>t1=l1^r1;Vector256<byte>t=t0|t1;returnt==Vector256<byte>.Zero;}}// Current implementation of SequenceEqualCore w/o length-check (already done)returnSequenceEqualCore(left,right);}

So it's more IL and more work for the JIT to do. Are these the reasons why it's done via [Intrinsic] directly in the JIT?

[EgorBo]

@gfoidl there are two issues with the managed approach:

1. We can only detect constant length for Span in late phases of JIT, so for IsKnownConstant case we'll have to carry a large tree through all phases. And yes, it needs extra support on JIT side for IsKnownConstant
2. I tried to do the same for String.Equals unrolling once and hit two issues: first, this happy-path creates a huge amount of locals so for some deep callsites we can stop tracking locals because of that (or stop inlining more into the graph). Second, inliner's budget problem. See Unroll String.Equals for constant input [0..16] length #64821

[IDisposable]

I know this is completely unrelated to the fix or changes being done, sorry... but has anyone ever tried reversing the methodName and className tests for a performance hack in the JIT itself?

[EgorBo]

@IDisposablelookupNamedIntrinsics never show up in our JIT traces so we don't bother. This code is only executed for methods with [Intrinsic] attribute so for 99% of methods it doesn't kick in.

We could use here a Trie/binary search if it was a real problem

[gfoidl]

@EgorBo thanks for the info, I understand.

[jakobbotsch]

Seems like you could just make this function return the first new node you added, since the call was replaced anyway, and have "normal" lowering proceed from there.

[EgorBo]

Good idea, done

[EgorBo]

@EgorBo thanks for the info, I understand.

Still, filed a PR #84002 to make it possible, so now you can use IsKnownConstant(span.Length)

[jakobbotsch]

It's a bit wasteful to go looking for the uses of this given that we know the arg they come from. E.g. you could do

Suggested change

	LIR::Use lArgUse;
	LIR::Use rArgUse;
	bool lFoundUse = BlockRange().TryGetUse(lArg, &lArgUse);
	bool rFoundUse = BlockRange().TryGetUse(rArg, &rArgUse);
	assert(lFoundUse && rFoundUse);
	CallArg* lArg = call->gtArgs.GetUserArgByIndex(0);
	GenTree*& lArgNode = lArg->GetLateNode() == nullptr ? lArg->EarlyNodeRef() : lArg->LateNodeRef();
	...
	LIR::Use lArgUse(BlockRange(), &lArgNode, call);

I don't have a super strong opinion on it.

[EgorBo]

Thank, will check in a follow up once SPMI is collected - want to see if it's worth the effort to improve this expansion. jit-diff utils found around 30 methods only

[EgorBo]

Failures are #83655 and #80619

[EgorBo]

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.SequenceEqual(data);boolTest2(Span<byte>data)=>data.StartsWith("test"u8);

Codegen diff: https://www.diffchecker.com/E1laymuB/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side.
Works for StartsWith.

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}publicstaticclassPaths{publicstaticReadOnlySpan<byte>Json=>"/json"u8;publicstaticReadOnlySpan<byte>Plaintext=>"/plaintext"u8;}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

(the difference should be bigger when SIMD is enabled)

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest(ReadOnlySpan<byte>data){return"hello world!"u8.SequenceEqual(data);}

Codegen diff: https://www.diffchecker.com/0VOpmvMj/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side (which is likely the case for e.g. StartsWith).

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

Author:	EgorBo
Assignees:	EgorBo
Labels:	area-CodeGen-coreclr
Milestone:	-

[gfoidl]

Can be dasm for Test1 be something like

G_M000_IG01: ;; offset=0000HG_M000_IG02: ;; offset=0000H 488B01 movrax, bword ptr [rcx] 8B5108 movedx, dword ptr [rcx+08H] 83FA0B cmpedx,117404je SHORT G_M000_IG04G_M000_IG03: ;; offset=000BH 33C0 xoreax,eax EB24 jmp SHORT G_M000_IG05G_M000_IG04: ;; offset=000FH 48BA68656C6C6F20776F movrdx,0x6F77206F6C6C6568483310xorrdx, qword ptr [rax] 48B96C6F20776F726C64 movrcx,0x646C726F77206F6C48334803xorrcx, qword ptr [rax+03H] 480BD1 orrdx,rcx 0F94C0 sete al 0FB6C0 movzxrax,alG_M000_IG05: ;; offset=0033H C3 ret; Total bytes of code 52

?
So that the "constant" LHS (here "hello world"u8) is read as long-constant instead of loaded from memory. Save two memory loads.

The assembly above is produced by this simple C# approach.

Code

usingSystem.Runtime.CompilerServices;usingSystem.Runtime.InteropServices;ReadOnlySpan<byte>test="hello world"u8;Console.WriteLine(Test1(test));
#if !DEBUGfor(inti=0;i<100;++i){if(i%10==0)Thread.Sleep(100);_=Test1(test);}
#endif
staticboolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.FastSequenceEqual(data);internalstaticclassMySpanExtensions{[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolFastSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr0=Unsafe.ReadUnaligned<long>(refrightRef);longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}thrownewNotSupportedException();}}

PS: the XOR-trick here is 👍🏻

[jakobbotsch]

Are you sure this is better than the naive version for ARM64 with CCMPs? What is the ARM64 codegen diff if you create AND(EQ(IND, IND), EQ(IND, IND)) instead?

[EgorBo]

The current codegen is (comparing 16 bytes):

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]CA030021 eor x1, x1, x3F9400000 ldr x0,[x0]F9400042 ldr x2,[x2]CA020000 eor x0, x0, x2AA000020 orr x0, x1, x0F100001F cmp x0, #09A9F17E0 cset x0, eq

cmp version presumably needs ifConversion path? Because here is what I see when I follow your suggestion:

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]EB03003F cmp x1, x39A9F17E1 cset x1, eqF9400000 ldr x0,[x0]F9400042 ldr x2,[x2]EB02001F cmp x0, x29A9F17E0 cset x0, eqEA00003F tst x1, x09A9F07E0 cset x0, ne

so we need to either do this opt in codegen or earlier for that. For me arm64 codegen doesn't look too bad, it's still better than not unrolled.

[jakobbotsch]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

[EgorBo]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

still doesn't want to convert to CCMP, IsInvariantInRange check fails, presumably because of IND side effects. Still, I think the current version is better than non-unrolled

[jakobbotsch]

You should be able to insert it in the right order so that there is no interference, e.g. probably

t0 = IND
t1 = IND
t2 = IND
t3 = IND
t4 = EQ(t0, t1)
t5 = EQ(t2, t3)
t6 = AND(t4, t5)

[jakobbotsch]

Although it's a bit odd there would be interference even with

t0 =IND
t1 =IND
t2 =EQ(t0, t1)
t3 =IND
t4 =IND
t5 =EQ(t3, t4)
t6 =AND(t2, t5)

Probably something I should take a look at.

[EgorBo]

but just in case I pushed a change to move all IND nodes to the front

[gfoidl]

I have a general question: why needs this to be done in JIT and not in managed code? Is it about throughput and / or IL-size?
In some areas there's native -> managed, here it's the opposite.
This is not a rant, just a question out of curiosity. And it makes contributing a bit harder (at least for me, as I don't know the internals of JIT very much).

To implement this in pure C# something like RuntimeHelpers.IsKnownConstant(ReadOnlySpan<byte>) is missing.
Assuming such a method exists, then it could look like:

Example managed implementation

[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(/* missing piece */RuntimeHelpers.IsKnownConstant(left)){if(len>=sizeof(int)&&len<=2*sizeof(int)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);intl0=Unsafe.ReadUnaligned<int>(refleftRef);intr0=Unsafe.ReadUnaligned<int>(refrightRef);intl1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refleftRef,len-sizeof(int)));intr1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refrightRef,len-sizeof(int)));intt0=l0^r0;intt1=l1^r1;intt=t0|t1;returnt==0;}if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longr0=Unsafe.ReadUnaligned<long>(refrightRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}if(Vector128.IsHardwareAccelerated&&len>=(uint)Vector128<byte>.Count&&len<=2*(uint)Vector128<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector128<byte>l0=Vector128.LoadUnsafe(refleftRef);Vector128<byte>r0=Vector128.LoadUnsafe(refrightRef);Vector128<byte>t0=l0^r0;Vector128<byte>l1=Vector128.LoadUnsafe(refleftRef,len-(uint)Vector128<byte>.Count);Vector128<byte>r1=Vector128.LoadUnsafe(refrightRef,len-(uint)Vector128<byte>.Count);Vector128<byte>t1=l1^r1;Vector128<byte>t=t0|t1;returnt==Vector128<byte>.Zero;}if(Vector256.IsHardwareAccelerated&&len>=(uint)Vector256<byte>.Count&&len<=2*(uint)Vector256<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector256<byte>l0=Vector256.LoadUnsafe(refleftRef);Vector256<byte>r0=Vector256.LoadUnsafe(refrightRef);Vector256<byte>t0=l0^r0;Vector256<byte>l1=Vector256.LoadUnsafe(refleftRef,len-(uint)Vector256<byte>.Count);Vector256<byte>r1=Vector256.LoadUnsafe(refrightRef,len-(uint)Vector256<byte>.Count);Vector256<byte>t1=l1^r1;Vector256<byte>t=t0|t1;returnt==Vector256<byte>.Zero;}}// Current implementation of SequenceEqualCore w/o length-check (already done)returnSequenceEqualCore(left,right);}

So it's more IL and more work for the JIT to do. Are these the reasons why it's done via [Intrinsic] directly in the JIT?

[EgorBo]

@gfoidl there are two issues with the managed approach:

1. We can only detect constant length for Span in late phases of JIT, so for IsKnownConstant case we'll have to carry a large tree through all phases. And yes, it needs extra support on JIT side for IsKnownConstant
2. I tried to do the same for String.Equals unrolling once and hit two issues: first, this happy-path creates a huge amount of locals so for some deep callsites we can stop tracking locals because of that (or stop inlining more into the graph). Second, inliner's budget problem. See Unroll String.Equals for constant input [0..16] length #64821

[IDisposable]

I know this is completely unrelated to the fix or changes being done, sorry... but has anyone ever tried reversing the methodName and className tests for a performance hack in the JIT itself?

[EgorBo]

@IDisposablelookupNamedIntrinsics never show up in our JIT traces so we don't bother. This code is only executed for methods with [Intrinsic] attribute so for 99% of methods it doesn't kick in.

We could use here a Trie/binary search if it was a real problem

[gfoidl]

@EgorBo thanks for the info, I understand.

[jakobbotsch]

Seems like you could just make this function return the first new node you added, since the call was replaced anyway, and have "normal" lowering proceed from there.

[EgorBo]

Good idea, done

[EgorBo]

@EgorBo thanks for the info, I understand.

Still, filed a PR #84002 to make it possible, so now you can use IsKnownConstant(span.Length)

[jakobbotsch]

It's a bit wasteful to go looking for the uses of this given that we know the arg they come from. E.g. you could do

Suggested change

	LIR::Use lArgUse;
	LIR::Use rArgUse;
	bool lFoundUse = BlockRange().TryGetUse(lArg, &lArgUse);
	bool rFoundUse = BlockRange().TryGetUse(rArg, &rArgUse);
	assert(lFoundUse && rFoundUse);
	CallArg* lArg = call->gtArgs.GetUserArgByIndex(0);
	GenTree*& lArgNode = lArg->GetLateNode() == nullptr ? lArg->EarlyNodeRef() : lArg->LateNodeRef();
	...
	LIR::Use lArgUse(BlockRange(), &lArgNode, call);

I don't have a super strong opinion on it.

[EgorBo]

Thank, will check in a follow up once SPMI is collected - want to see if it's worth the effort to improve this expansion. jit-diff utils found around 30 methods only

[EgorBo]

Failures are #83655 and #80619

[EgorBo]

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.SequenceEqual(data);boolTest2(Span<byte>data)=>data.StartsWith("test"u8);

Codegen diff: https://www.diffchecker.com/E1laymuB/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side.
Works for StartsWith.

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}publicstaticclassPaths{publicstaticReadOnlySpan<byte>Json=>"/json"u8;publicstaticReadOnlySpan<byte>Plaintext=>"/plaintext"u8;}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

(the difference should be bigger when SIMD is enabled)

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest(ReadOnlySpan<byte>data){return"hello world!"u8.SequenceEqual(data);}

Codegen diff: https://www.diffchecker.com/0VOpmvMj/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side (which is likely the case for e.g. StartsWith).

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

Author:	EgorBo
Assignees:	EgorBo
Labels:	area-CodeGen-coreclr
Milestone:	-

[gfoidl]

Can be dasm for Test1 be something like

G_M000_IG01: ;; offset=0000HG_M000_IG02: ;; offset=0000H 488B01 movrax, bword ptr [rcx] 8B5108 movedx, dword ptr [rcx+08H] 83FA0B cmpedx,117404je SHORT G_M000_IG04G_M000_IG03: ;; offset=000BH 33C0 xoreax,eax EB24 jmp SHORT G_M000_IG05G_M000_IG04: ;; offset=000FH 48BA68656C6C6F20776F movrdx,0x6F77206F6C6C6568483310xorrdx, qword ptr [rax] 48B96C6F20776F726C64 movrcx,0x646C726F77206F6C48334803xorrcx, qword ptr [rax+03H] 480BD1 orrdx,rcx 0F94C0 sete al 0FB6C0 movzxrax,alG_M000_IG05: ;; offset=0033H C3 ret; Total bytes of code 52

?
So that the "constant" LHS (here "hello world"u8) is read as long-constant instead of loaded from memory. Save two memory loads.

The assembly above is produced by this simple C# approach.

Code

usingSystem.Runtime.CompilerServices;usingSystem.Runtime.InteropServices;ReadOnlySpan<byte>test="hello world"u8;Console.WriteLine(Test1(test));
#if !DEBUGfor(inti=0;i<100;++i){if(i%10==0)Thread.Sleep(100);_=Test1(test);}
#endif
staticboolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.FastSequenceEqual(data);internalstaticclassMySpanExtensions{[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolFastSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr0=Unsafe.ReadUnaligned<long>(refrightRef);longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}thrownewNotSupportedException();}}

PS: the XOR-trick here is 👍🏻

[jakobbotsch]

Are you sure this is better than the naive version for ARM64 with CCMPs? What is the ARM64 codegen diff if you create AND(EQ(IND, IND), EQ(IND, IND)) instead?

[EgorBo]

The current codegen is (comparing 16 bytes):

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]CA030021 eor x1, x1, x3F9400000 ldr x0,[x0]F9400042 ldr x2,[x2]CA020000 eor x0, x0, x2AA000020 orr x0, x1, x0F100001F cmp x0, #09A9F17E0 cset x0, eq

cmp version presumably needs ifConversion path? Because here is what I see when I follow your suggestion:

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]EB03003F cmp x1, x39A9F17E1 cset x1, eqF9400000 ldr x0,[x0]F9400042 ldr x2,[x2]EB02001F cmp x0, x29A9F17E0 cset x0, eqEA00003F tst x1, x09A9F07E0 cset x0, ne

so we need to either do this opt in codegen or earlier for that. For me arm64 codegen doesn't look too bad, it's still better than not unrolled.

[jakobbotsch]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

[EgorBo]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

still doesn't want to convert to CCMP, IsInvariantInRange check fails, presumably because of IND side effects. Still, I think the current version is better than non-unrolled

[jakobbotsch]

You should be able to insert it in the right order so that there is no interference, e.g. probably

t0 = IND
t1 = IND
t2 = IND
t3 = IND
t4 = EQ(t0, t1)
t5 = EQ(t2, t3)
t6 = AND(t4, t5)

[jakobbotsch]

Although it's a bit odd there would be interference even with

t0 =IND
t1 =IND
t2 =EQ(t0, t1)
t3 =IND
t4 =IND
t5 =EQ(t3, t4)
t6 =AND(t2, t5)

Probably something I should take a look at.

[EgorBo]

but just in case I pushed a change to move all IND nodes to the front

[gfoidl]

I have a general question: why needs this to be done in JIT and not in managed code? Is it about throughput and / or IL-size?
In some areas there's native -> managed, here it's the opposite.
This is not a rant, just a question out of curiosity. And it makes contributing a bit harder (at least for me, as I don't know the internals of JIT very much).

To implement this in pure C# something like RuntimeHelpers.IsKnownConstant(ReadOnlySpan<byte>) is missing.
Assuming such a method exists, then it could look like:

Example managed implementation

[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(/* missing piece */RuntimeHelpers.IsKnownConstant(left)){if(len>=sizeof(int)&&len<=2*sizeof(int)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);intl0=Unsafe.ReadUnaligned<int>(refleftRef);intr0=Unsafe.ReadUnaligned<int>(refrightRef);intl1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refleftRef,len-sizeof(int)));intr1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refrightRef,len-sizeof(int)));intt0=l0^r0;intt1=l1^r1;intt=t0|t1;returnt==0;}if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longr0=Unsafe.ReadUnaligned<long>(refrightRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}if(Vector128.IsHardwareAccelerated&&len>=(uint)Vector128<byte>.Count&&len<=2*(uint)Vector128<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector128<byte>l0=Vector128.LoadUnsafe(refleftRef);Vector128<byte>r0=Vector128.LoadUnsafe(refrightRef);Vector128<byte>t0=l0^r0;Vector128<byte>l1=Vector128.LoadUnsafe(refleftRef,len-(uint)Vector128<byte>.Count);Vector128<byte>r1=Vector128.LoadUnsafe(refrightRef,len-(uint)Vector128<byte>.Count);Vector128<byte>t1=l1^r1;Vector128<byte>t=t0|t1;returnt==Vector128<byte>.Zero;}if(Vector256.IsHardwareAccelerated&&len>=(uint)Vector256<byte>.Count&&len<=2*(uint)Vector256<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector256<byte>l0=Vector256.LoadUnsafe(refleftRef);Vector256<byte>r0=Vector256.LoadUnsafe(refrightRef);Vector256<byte>t0=l0^r0;Vector256<byte>l1=Vector256.LoadUnsafe(refleftRef,len-(uint)Vector256<byte>.Count);Vector256<byte>r1=Vector256.LoadUnsafe(refrightRef,len-(uint)Vector256<byte>.Count);Vector256<byte>t1=l1^r1;Vector256<byte>t=t0|t1;returnt==Vector256<byte>.Zero;}}// Current implementation of SequenceEqualCore w/o length-check (already done)returnSequenceEqualCore(left,right);}

So it's more IL and more work for the JIT to do. Are these the reasons why it's done via [Intrinsic] directly in the JIT?

[EgorBo]

@gfoidl there are two issues with the managed approach:

1. We can only detect constant length for Span in late phases of JIT, so for IsKnownConstant case we'll have to carry a large tree through all phases. And yes, it needs extra support on JIT side for IsKnownConstant
2. I tried to do the same for String.Equals unrolling once and hit two issues: first, this happy-path creates a huge amount of locals so for some deep callsites we can stop tracking locals because of that (or stop inlining more into the graph). Second, inliner's budget problem. See Unroll String.Equals for constant input [0..16] length #64821

[IDisposable]

I know this is completely unrelated to the fix or changes being done, sorry... but has anyone ever tried reversing the methodName and className tests for a performance hack in the JIT itself?

[EgorBo]

@IDisposablelookupNamedIntrinsics never show up in our JIT traces so we don't bother. This code is only executed for methods with [Intrinsic] attribute so for 99% of methods it doesn't kick in.

We could use here a Trie/binary search if it was a real problem

[gfoidl]

@EgorBo thanks for the info, I understand.

[jakobbotsch]

Seems like you could just make this function return the first new node you added, since the call was replaced anyway, and have "normal" lowering proceed from there.

[EgorBo]

Good idea, done

[EgorBo]

@EgorBo thanks for the info, I understand.

Still, filed a PR #84002 to make it possible, so now you can use IsKnownConstant(span.Length)

[jakobbotsch]

It's a bit wasteful to go looking for the uses of this given that we know the arg they come from. E.g. you could do

Suggested change

	LIR::Use lArgUse;
	LIR::Use rArgUse;
	bool lFoundUse = BlockRange().TryGetUse(lArg, &lArgUse);
	bool rFoundUse = BlockRange().TryGetUse(rArg, &rArgUse);
	assert(lFoundUse && rFoundUse);
	CallArg* lArg = call->gtArgs.GetUserArgByIndex(0);
	GenTree*& lArgNode = lArg->GetLateNode() == nullptr ? lArg->EarlyNodeRef() : lArg->LateNodeRef();
	...
	LIR::Use lArgUse(BlockRange(), &lArgNode, call);

I don't have a super strong opinion on it.

[EgorBo]

Thank, will check in a follow up once SPMI is collected - want to see if it's worth the effort to improve this expansion. jit-diff utils found around 30 methods only

[EgorBo]

Failures are #83655 and #80619

[EgorBo]

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.SequenceEqual(data);boolTest2(Span<byte>data)=>data.StartsWith("test"u8);

Codegen diff: https://www.diffchecker.com/E1laymuB/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side.
Works for StartsWith.

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}publicstaticclassPaths{publicstaticReadOnlySpan<byte>Json=>"/json"u8;publicstaticReadOnlySpan<byte>Plaintext=>"/plaintext"u8;}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

(the difference should be bigger when SIMD is enabled)

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest(ReadOnlySpan<byte>data){return"hello world!"u8.SequenceEqual(data);}

Codegen diff: https://www.diffchecker.com/0VOpmvMj/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side (which is likely the case for e.g. StartsWith).

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

Author:	EgorBo
Assignees:	EgorBo
Labels:	area-CodeGen-coreclr
Milestone:	-

[gfoidl]

Can be dasm for Test1 be something like

G_M000_IG01: ;; offset=0000HG_M000_IG02: ;; offset=0000H 488B01 movrax, bword ptr [rcx] 8B5108 movedx, dword ptr [rcx+08H] 83FA0B cmpedx,117404je SHORT G_M000_IG04G_M000_IG03: ;; offset=000BH 33C0 xoreax,eax EB24 jmp SHORT G_M000_IG05G_M000_IG04: ;; offset=000FH 48BA68656C6C6F20776F movrdx,0x6F77206F6C6C6568483310xorrdx, qword ptr [rax] 48B96C6F20776F726C64 movrcx,0x646C726F77206F6C48334803xorrcx, qword ptr [rax+03H] 480BD1 orrdx,rcx 0F94C0 sete al 0FB6C0 movzxrax,alG_M000_IG05: ;; offset=0033H C3 ret; Total bytes of code 52

?
So that the "constant" LHS (here "hello world"u8) is read as long-constant instead of loaded from memory. Save two memory loads.

The assembly above is produced by this simple C# approach.

Code

usingSystem.Runtime.CompilerServices;usingSystem.Runtime.InteropServices;ReadOnlySpan<byte>test="hello world"u8;Console.WriteLine(Test1(test));
#if !DEBUGfor(inti=0;i<100;++i){if(i%10==0)Thread.Sleep(100);_=Test1(test);}
#endif
staticboolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.FastSequenceEqual(data);internalstaticclassMySpanExtensions{[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolFastSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr0=Unsafe.ReadUnaligned<long>(refrightRef);longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}thrownewNotSupportedException();}}

PS: the XOR-trick here is 👍🏻

[jakobbotsch]

Are you sure this is better than the naive version for ARM64 with CCMPs? What is the ARM64 codegen diff if you create AND(EQ(IND, IND), EQ(IND, IND)) instead?

[EgorBo]

The current codegen is (comparing 16 bytes):

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]CA030021 eor x1, x1, x3F9400000 ldr x0,[x0]F9400042 ldr x2,[x2]CA020000 eor x0, x0, x2AA000020 orr x0, x1, x0F100001F cmp x0, #09A9F17E0 cset x0, eq

cmp version presumably needs ifConversion path? Because here is what I see when I follow your suggestion:

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]EB03003F cmp x1, x39A9F17E1 cset x1, eqF9400000 ldr x0,[x0]F9400042 ldr x2,[x2]EB02001F cmp x0, x29A9F17E0 cset x0, eqEA00003F tst x1, x09A9F07E0 cset x0, ne

so we need to either do this opt in codegen or earlier for that. For me arm64 codegen doesn't look too bad, it's still better than not unrolled.

[jakobbotsch]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

[EgorBo]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

still doesn't want to convert to CCMP, IsInvariantInRange check fails, presumably because of IND side effects. Still, I think the current version is better than non-unrolled

[jakobbotsch]

You should be able to insert it in the right order so that there is no interference, e.g. probably

t0 = IND
t1 = IND
t2 = IND
t3 = IND
t4 = EQ(t0, t1)
t5 = EQ(t2, t3)
t6 = AND(t4, t5)

[jakobbotsch]

Although it's a bit odd there would be interference even with

t0 =IND
t1 =IND
t2 =EQ(t0, t1)
t3 =IND
t4 =IND
t5 =EQ(t3, t4)
t6 =AND(t2, t5)

Probably something I should take a look at.

[EgorBo]

but just in case I pushed a change to move all IND nodes to the front

[gfoidl]

I have a general question: why needs this to be done in JIT and not in managed code? Is it about throughput and / or IL-size?
In some areas there's native -> managed, here it's the opposite.
This is not a rant, just a question out of curiosity. And it makes contributing a bit harder (at least for me, as I don't know the internals of JIT very much).

To implement this in pure C# something like RuntimeHelpers.IsKnownConstant(ReadOnlySpan<byte>) is missing.
Assuming such a method exists, then it could look like:

Example managed implementation

[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(/* missing piece */RuntimeHelpers.IsKnownConstant(left)){if(len>=sizeof(int)&&len<=2*sizeof(int)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);intl0=Unsafe.ReadUnaligned<int>(refleftRef);intr0=Unsafe.ReadUnaligned<int>(refrightRef);intl1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refleftRef,len-sizeof(int)));intr1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refrightRef,len-sizeof(int)));intt0=l0^r0;intt1=l1^r1;intt=t0|t1;returnt==0;}if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longr0=Unsafe.ReadUnaligned<long>(refrightRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}if(Vector128.IsHardwareAccelerated&&len>=(uint)Vector128<byte>.Count&&len<=2*(uint)Vector128<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector128<byte>l0=Vector128.LoadUnsafe(refleftRef);Vector128<byte>r0=Vector128.LoadUnsafe(refrightRef);Vector128<byte>t0=l0^r0;Vector128<byte>l1=Vector128.LoadUnsafe(refleftRef,len-(uint)Vector128<byte>.Count);Vector128<byte>r1=Vector128.LoadUnsafe(refrightRef,len-(uint)Vector128<byte>.Count);Vector128<byte>t1=l1^r1;Vector128<byte>t=t0|t1;returnt==Vector128<byte>.Zero;}if(Vector256.IsHardwareAccelerated&&len>=(uint)Vector256<byte>.Count&&len<=2*(uint)Vector256<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector256<byte>l0=Vector256.LoadUnsafe(refleftRef);Vector256<byte>r0=Vector256.LoadUnsafe(refrightRef);Vector256<byte>t0=l0^r0;Vector256<byte>l1=Vector256.LoadUnsafe(refleftRef,len-(uint)Vector256<byte>.Count);Vector256<byte>r1=Vector256.LoadUnsafe(refrightRef,len-(uint)Vector256<byte>.Count);Vector256<byte>t1=l1^r1;Vector256<byte>t=t0|t1;returnt==Vector256<byte>.Zero;}}// Current implementation of SequenceEqualCore w/o length-check (already done)returnSequenceEqualCore(left,right);}

So it's more IL and more work for the JIT to do. Are these the reasons why it's done via [Intrinsic] directly in the JIT?

[EgorBo]

@gfoidl there are two issues with the managed approach:

1. We can only detect constant length for Span in late phases of JIT, so for IsKnownConstant case we'll have to carry a large tree through all phases. And yes, it needs extra support on JIT side for IsKnownConstant
2. I tried to do the same for String.Equals unrolling once and hit two issues: first, this happy-path creates a huge amount of locals so for some deep callsites we can stop tracking locals because of that (or stop inlining more into the graph). Second, inliner's budget problem. See Unroll String.Equals for constant input [0..16] length #64821

[IDisposable]

I know this is completely unrelated to the fix or changes being done, sorry... but has anyone ever tried reversing the methodName and className tests for a performance hack in the JIT itself?

[EgorBo]

@IDisposablelookupNamedIntrinsics never show up in our JIT traces so we don't bother. This code is only executed for methods with [Intrinsic] attribute so for 99% of methods it doesn't kick in.

We could use here a Trie/binary search if it was a real problem

[gfoidl]

@EgorBo thanks for the info, I understand.

[jakobbotsch]

Seems like you could just make this function return the first new node you added, since the call was replaced anyway, and have "normal" lowering proceed from there.

[EgorBo]

Good idea, done

[EgorBo]

@EgorBo thanks for the info, I understand.

Still, filed a PR #84002 to make it possible, so now you can use IsKnownConstant(span.Length)

[jakobbotsch]

It's a bit wasteful to go looking for the uses of this given that we know the arg they come from. E.g. you could do

Suggested change

	LIR::Use lArgUse;
	LIR::Use rArgUse;
	bool lFoundUse = BlockRange().TryGetUse(lArg, &lArgUse);
	bool rFoundUse = BlockRange().TryGetUse(rArg, &rArgUse);
	assert(lFoundUse && rFoundUse);
	CallArg* lArg = call->gtArgs.GetUserArgByIndex(0);
	GenTree*& lArgNode = lArg->GetLateNode() == nullptr ? lArg->EarlyNodeRef() : lArg->LateNodeRef();
	...
	LIR::Use lArgUse(BlockRange(), &lArgNode, call);

I don't have a super strong opinion on it.

[EgorBo]

Thank, will check in a follow up once SPMI is collected - want to see if it's worth the effort to improve this expansion. jit-diff utils found around 30 methods only

[EgorBo]

Failures are #83655 and #80619

[EgorBo]

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.SequenceEqual(data);boolTest2(Span<byte>data)=>data.StartsWith("test"u8);

Codegen diff: https://www.diffchecker.com/E1laymuB/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side.
Works for StartsWith.

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}publicstaticclassPaths{publicstaticReadOnlySpan<byte>Json=>"/json"u8;publicstaticReadOnlySpan<byte>Plaintext=>"/plaintext"u8;}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

(the difference should be bigger when SIMD is enabled)

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest(ReadOnlySpan<byte>data){return"hello world!"u8.SequenceEqual(data);}

Codegen diff: https://www.diffchecker.com/0VOpmvMj/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side (which is likely the case for e.g. StartsWith).

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

Author:	EgorBo
Assignees:	EgorBo
Labels:	area-CodeGen-coreclr
Milestone:	-

[gfoidl]

Can be dasm for Test1 be something like

G_M000_IG01: ;; offset=0000HG_M000_IG02: ;; offset=0000H 488B01 movrax, bword ptr [rcx] 8B5108 movedx, dword ptr [rcx+08H] 83FA0B cmpedx,117404je SHORT G_M000_IG04G_M000_IG03: ;; offset=000BH 33C0 xoreax,eax EB24 jmp SHORT G_M000_IG05G_M000_IG04: ;; offset=000FH 48BA68656C6C6F20776F movrdx,0x6F77206F6C6C6568483310xorrdx, qword ptr [rax] 48B96C6F20776F726C64 movrcx,0x646C726F77206F6C48334803xorrcx, qword ptr [rax+03H] 480BD1 orrdx,rcx 0F94C0 sete al 0FB6C0 movzxrax,alG_M000_IG05: ;; offset=0033H C3 ret; Total bytes of code 52

?
So that the "constant" LHS (here "hello world"u8) is read as long-constant instead of loaded from memory. Save two memory loads.

The assembly above is produced by this simple C# approach.

Code

usingSystem.Runtime.CompilerServices;usingSystem.Runtime.InteropServices;ReadOnlySpan<byte>test="hello world"u8;Console.WriteLine(Test1(test));
#if !DEBUGfor(inti=0;i<100;++i){if(i%10==0)Thread.Sleep(100);_=Test1(test);}
#endif
staticboolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.FastSequenceEqual(data);internalstaticclassMySpanExtensions{[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolFastSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr0=Unsafe.ReadUnaligned<long>(refrightRef);longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}thrownewNotSupportedException();}}

PS: the XOR-trick here is 👍🏻

[jakobbotsch]

Are you sure this is better than the naive version for ARM64 with CCMPs? What is the ARM64 codegen diff if you create AND(EQ(IND, IND), EQ(IND, IND)) instead?

[EgorBo]

The current codegen is (comparing 16 bytes):

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]CA030021 eor x1, x1, x3F9400000 ldr x0,[x0]F9400042 ldr x2,[x2]CA020000 eor x0, x0, x2AA000020 orr x0, x1, x0F100001F cmp x0, #09A9F17E0 cset x0, eq

cmp version presumably needs ifConversion path? Because here is what I see when I follow your suggestion:

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]EB03003F cmp x1, x39A9F17E1 cset x1, eqF9400000 ldr x0,[x0]F9400042 ldr x2,[x2]EB02001F cmp x0, x29A9F17E0 cset x0, eqEA00003F tst x1, x09A9F07E0 cset x0, ne

so we need to either do this opt in codegen or earlier for that. For me arm64 codegen doesn't look too bad, it's still better than not unrolled.

[jakobbotsch]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

[EgorBo]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

still doesn't want to convert to CCMP, IsInvariantInRange check fails, presumably because of IND side effects. Still, I think the current version is better than non-unrolled

[jakobbotsch]

You should be able to insert it in the right order so that there is no interference, e.g. probably

t0 = IND
t1 = IND
t2 = IND
t3 = IND
t4 = EQ(t0, t1)
t5 = EQ(t2, t3)
t6 = AND(t4, t5)

[jakobbotsch]

Although it's a bit odd there would be interference even with

t0 =IND
t1 =IND
t2 =EQ(t0, t1)
t3 =IND
t4 =IND
t5 =EQ(t3, t4)
t6 =AND(t2, t5)

Probably something I should take a look at.

[EgorBo]

but just in case I pushed a change to move all IND nodes to the front

[gfoidl]

I have a general question: why needs this to be done in JIT and not in managed code? Is it about throughput and / or IL-size?
In some areas there's native -> managed, here it's the opposite.
This is not a rant, just a question out of curiosity. And it makes contributing a bit harder (at least for me, as I don't know the internals of JIT very much).

To implement this in pure C# something like RuntimeHelpers.IsKnownConstant(ReadOnlySpan<byte>) is missing.
Assuming such a method exists, then it could look like:

Example managed implementation

[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(/* missing piece */RuntimeHelpers.IsKnownConstant(left)){if(len>=sizeof(int)&&len<=2*sizeof(int)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);intl0=Unsafe.ReadUnaligned<int>(refleftRef);intr0=Unsafe.ReadUnaligned<int>(refrightRef);intl1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refleftRef,len-sizeof(int)));intr1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refrightRef,len-sizeof(int)));intt0=l0^r0;intt1=l1^r1;intt=t0|t1;returnt==0;}if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longr0=Unsafe.ReadUnaligned<long>(refrightRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}if(Vector128.IsHardwareAccelerated&&len>=(uint)Vector128<byte>.Count&&len<=2*(uint)Vector128<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector128<byte>l0=Vector128.LoadUnsafe(refleftRef);Vector128<byte>r0=Vector128.LoadUnsafe(refrightRef);Vector128<byte>t0=l0^r0;Vector128<byte>l1=Vector128.LoadUnsafe(refleftRef,len-(uint)Vector128<byte>.Count);Vector128<byte>r1=Vector128.LoadUnsafe(refrightRef,len-(uint)Vector128<byte>.Count);Vector128<byte>t1=l1^r1;Vector128<byte>t=t0|t1;returnt==Vector128<byte>.Zero;}if(Vector256.IsHardwareAccelerated&&len>=(uint)Vector256<byte>.Count&&len<=2*(uint)Vector256<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector256<byte>l0=Vector256.LoadUnsafe(refleftRef);Vector256<byte>r0=Vector256.LoadUnsafe(refrightRef);Vector256<byte>t0=l0^r0;Vector256<byte>l1=Vector256.LoadUnsafe(refleftRef,len-(uint)Vector256<byte>.Count);Vector256<byte>r1=Vector256.LoadUnsafe(refrightRef,len-(uint)Vector256<byte>.Count);Vector256<byte>t1=l1^r1;Vector256<byte>t=t0|t1;returnt==Vector256<byte>.Zero;}}// Current implementation of SequenceEqualCore w/o length-check (already done)returnSequenceEqualCore(left,right);}

So it's more IL and more work for the JIT to do. Are these the reasons why it's done via [Intrinsic] directly in the JIT?

[EgorBo]

@gfoidl there are two issues with the managed approach:

1. We can only detect constant length for Span in late phases of JIT, so for IsKnownConstant case we'll have to carry a large tree through all phases. And yes, it needs extra support on JIT side for IsKnownConstant
2. I tried to do the same for String.Equals unrolling once and hit two issues: first, this happy-path creates a huge amount of locals so for some deep callsites we can stop tracking locals because of that (or stop inlining more into the graph). Second, inliner's budget problem. See Unroll String.Equals for constant input [0..16] length #64821

[IDisposable]

I know this is completely unrelated to the fix or changes being done, sorry... but has anyone ever tried reversing the methodName and className tests for a performance hack in the JIT itself?

[EgorBo]

@IDisposablelookupNamedIntrinsics never show up in our JIT traces so we don't bother. This code is only executed for methods with [Intrinsic] attribute so for 99% of methods it doesn't kick in.

We could use here a Trie/binary search if it was a real problem

[gfoidl]

@EgorBo thanks for the info, I understand.

[jakobbotsch]

Seems like you could just make this function return the first new node you added, since the call was replaced anyway, and have "normal" lowering proceed from there.

[EgorBo]

Good idea, done

[EgorBo]

@EgorBo thanks for the info, I understand.

Still, filed a PR #84002 to make it possible, so now you can use IsKnownConstant(span.Length)

[jakobbotsch]

It's a bit wasteful to go looking for the uses of this given that we know the arg they come from. E.g. you could do

Suggested change

	LIR::Use lArgUse;
	LIR::Use rArgUse;
	bool lFoundUse = BlockRange().TryGetUse(lArg, &lArgUse);
	bool rFoundUse = BlockRange().TryGetUse(rArg, &rArgUse);
	assert(lFoundUse && rFoundUse);
	CallArg* lArg = call->gtArgs.GetUserArgByIndex(0);
	GenTree*& lArgNode = lArg->GetLateNode() == nullptr ? lArg->EarlyNodeRef() : lArg->LateNodeRef();
	...
	LIR::Use lArgUse(BlockRange(), &lArgNode, call);

I don't have a super strong opinion on it.

[EgorBo]

Thank, will check in a follow up once SPMI is collected - want to see if it's worth the effort to improve this expansion. jit-diff utils found around 30 methods only

[EgorBo]

Failures are #83655 and #80619

[EgorBo]

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.SequenceEqual(data);boolTest2(Span<byte>data)=>data.StartsWith("test"u8);

Codegen diff: https://www.diffchecker.com/E1laymuB/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side.
Works for StartsWith.

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}publicstaticclassPaths{publicstaticReadOnlySpan<byte>Json=>"/json"u8;publicstaticReadOnlySpan<byte>Plaintext=>"/plaintext"u8;}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

(the difference should be bigger when SIMD is enabled)

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest(ReadOnlySpan<byte>data){return"hello world!"u8.SequenceEqual(data);}

Codegen diff: https://www.diffchecker.com/0VOpmvMj/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side (which is likely the case for e.g. StartsWith).

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

Author:	EgorBo
Assignees:	EgorBo
Labels:	area-CodeGen-coreclr
Milestone:	-

[gfoidl]

Can be dasm for Test1 be something like

G_M000_IG01: ;; offset=0000HG_M000_IG02: ;; offset=0000H 488B01 movrax, bword ptr [rcx] 8B5108 movedx, dword ptr [rcx+08H] 83FA0B cmpedx,117404je SHORT G_M000_IG04G_M000_IG03: ;; offset=000BH 33C0 xoreax,eax EB24 jmp SHORT G_M000_IG05G_M000_IG04: ;; offset=000FH 48BA68656C6C6F20776F movrdx,0x6F77206F6C6C6568483310xorrdx, qword ptr [rax] 48B96C6F20776F726C64 movrcx,0x646C726F77206F6C48334803xorrcx, qword ptr [rax+03H] 480BD1 orrdx,rcx 0F94C0 sete al 0FB6C0 movzxrax,alG_M000_IG05: ;; offset=0033H C3 ret; Total bytes of code 52

?
So that the "constant" LHS (here "hello world"u8) is read as long-constant instead of loaded from memory. Save two memory loads.

The assembly above is produced by this simple C# approach.

Code

usingSystem.Runtime.CompilerServices;usingSystem.Runtime.InteropServices;ReadOnlySpan<byte>test="hello world"u8;Console.WriteLine(Test1(test));
#if !DEBUGfor(inti=0;i<100;++i){if(i%10==0)Thread.Sleep(100);_=Test1(test);}
#endif
staticboolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.FastSequenceEqual(data);internalstaticclassMySpanExtensions{[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolFastSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr0=Unsafe.ReadUnaligned<long>(refrightRef);longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}thrownewNotSupportedException();}}

PS: the XOR-trick here is 👍🏻

[jakobbotsch]

Are you sure this is better than the naive version for ARM64 with CCMPs? What is the ARM64 codegen diff if you create AND(EQ(IND, IND), EQ(IND, IND)) instead?

[EgorBo]

The current codegen is (comparing 16 bytes):

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]CA030021 eor x1, x1, x3F9400000 ldr x0,[x0]F9400042 ldr x2,[x2]CA020000 eor x0, x0, x2AA000020 orr x0, x1, x0F100001F cmp x0, #09A9F17E0 cset x0, eq

cmp version presumably needs ifConversion path? Because here is what I see when I follow your suggestion:

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]EB03003F cmp x1, x39A9F17E1 cset x1, eqF9400000 ldr x0,[x0]F9400042 ldr x2,[x2]EB02001F cmp x0, x29A9F17E0 cset x0, eqEA00003F tst x1, x09A9F07E0 cset x0, ne

so we need to either do this opt in codegen or earlier for that. For me arm64 codegen doesn't look too bad, it's still better than not unrolled.

[jakobbotsch]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

[EgorBo]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

still doesn't want to convert to CCMP, IsInvariantInRange check fails, presumably because of IND side effects. Still, I think the current version is better than non-unrolled

[jakobbotsch]

You should be able to insert it in the right order so that there is no interference, e.g. probably

t0 = IND
t1 = IND
t2 = IND
t3 = IND
t4 = EQ(t0, t1)
t5 = EQ(t2, t3)
t6 = AND(t4, t5)

[jakobbotsch]

Although it's a bit odd there would be interference even with

t0 =IND
t1 =IND
t2 =EQ(t0, t1)
t3 =IND
t4 =IND
t5 =EQ(t3, t4)
t6 =AND(t2, t5)

Probably something I should take a look at.

[EgorBo]

but just in case I pushed a change to move all IND nodes to the front

[gfoidl]

I have a general question: why needs this to be done in JIT and not in managed code? Is it about throughput and / or IL-size?
In some areas there's native -> managed, here it's the opposite.
This is not a rant, just a question out of curiosity. And it makes contributing a bit harder (at least for me, as I don't know the internals of JIT very much).

To implement this in pure C# something like RuntimeHelpers.IsKnownConstant(ReadOnlySpan<byte>) is missing.
Assuming such a method exists, then it could look like:

Example managed implementation

[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(/* missing piece */RuntimeHelpers.IsKnownConstant(left)){if(len>=sizeof(int)&&len<=2*sizeof(int)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);intl0=Unsafe.ReadUnaligned<int>(refleftRef);intr0=Unsafe.ReadUnaligned<int>(refrightRef);intl1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refleftRef,len-sizeof(int)));intr1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refrightRef,len-sizeof(int)));intt0=l0^r0;intt1=l1^r1;intt=t0|t1;returnt==0;}if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longr0=Unsafe.ReadUnaligned<long>(refrightRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}if(Vector128.IsHardwareAccelerated&&len>=(uint)Vector128<byte>.Count&&len<=2*(uint)Vector128<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector128<byte>l0=Vector128.LoadUnsafe(refleftRef);Vector128<byte>r0=Vector128.LoadUnsafe(refrightRef);Vector128<byte>t0=l0^r0;Vector128<byte>l1=Vector128.LoadUnsafe(refleftRef,len-(uint)Vector128<byte>.Count);Vector128<byte>r1=Vector128.LoadUnsafe(refrightRef,len-(uint)Vector128<byte>.Count);Vector128<byte>t1=l1^r1;Vector128<byte>t=t0|t1;returnt==Vector128<byte>.Zero;}if(Vector256.IsHardwareAccelerated&&len>=(uint)Vector256<byte>.Count&&len<=2*(uint)Vector256<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector256<byte>l0=Vector256.LoadUnsafe(refleftRef);Vector256<byte>r0=Vector256.LoadUnsafe(refrightRef);Vector256<byte>t0=l0^r0;Vector256<byte>l1=Vector256.LoadUnsafe(refleftRef,len-(uint)Vector256<byte>.Count);Vector256<byte>r1=Vector256.LoadUnsafe(refrightRef,len-(uint)Vector256<byte>.Count);Vector256<byte>t1=l1^r1;Vector256<byte>t=t0|t1;returnt==Vector256<byte>.Zero;}}// Current implementation of SequenceEqualCore w/o length-check (already done)returnSequenceEqualCore(left,right);}

So it's more IL and more work for the JIT to do. Are these the reasons why it's done via [Intrinsic] directly in the JIT?

[EgorBo]

@gfoidl there are two issues with the managed approach:

1. We can only detect constant length for Span in late phases of JIT, so for IsKnownConstant case we'll have to carry a large tree through all phases. And yes, it needs extra support on JIT side for IsKnownConstant
2. I tried to do the same for String.Equals unrolling once and hit two issues: first, this happy-path creates a huge amount of locals so for some deep callsites we can stop tracking locals because of that (or stop inlining more into the graph). Second, inliner's budget problem. See Unroll String.Equals for constant input [0..16] length #64821

[IDisposable]

I know this is completely unrelated to the fix or changes being done, sorry... but has anyone ever tried reversing the methodName and className tests for a performance hack in the JIT itself?

[EgorBo]

@IDisposablelookupNamedIntrinsics never show up in our JIT traces so we don't bother. This code is only executed for methods with [Intrinsic] attribute so for 99% of methods it doesn't kick in.

We could use here a Trie/binary search if it was a real problem

[gfoidl]

@EgorBo thanks for the info, I understand.

[jakobbotsch]

Seems like you could just make this function return the first new node you added, since the call was replaced anyway, and have "normal" lowering proceed from there.

[EgorBo]

Good idea, done

[EgorBo]

@EgorBo thanks for the info, I understand.

Still, filed a PR #84002 to make it possible, so now you can use IsKnownConstant(span.Length)

[jakobbotsch]

It's a bit wasteful to go looking for the uses of this given that we know the arg they come from. E.g. you could do

Suggested change

	LIR::Use lArgUse;
	LIR::Use rArgUse;
	bool lFoundUse = BlockRange().TryGetUse(lArg, &lArgUse);
	bool rFoundUse = BlockRange().TryGetUse(rArg, &rArgUse);
	assert(lFoundUse && rFoundUse);
	CallArg* lArg = call->gtArgs.GetUserArgByIndex(0);
	GenTree*& lArgNode = lArg->GetLateNode() == nullptr ? lArg->EarlyNodeRef() : lArg->LateNodeRef();
	...
	LIR::Use lArgUse(BlockRange(), &lArgNode, call);

I don't have a super strong opinion on it.

[EgorBo]

Thank, will check in a follow up once SPMI is collected - want to see if it's worth the effort to improve this expansion. jit-diff utils found around 30 methods only

[EgorBo]

Failures are #83655 and #80619

[EgorBo]

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.SequenceEqual(data);boolTest2(Span<byte>data)=>data.StartsWith("test"u8);

Codegen diff: https://www.diffchecker.com/E1laymuB/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side.
Works for StartsWith.

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}publicstaticclassPaths{publicstaticReadOnlySpan<byte>Json=>"/json"u8;publicstaticReadOnlySpan<byte>Plaintext=>"/plaintext"u8;}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

(the difference should be bigger when SIMD is enabled)

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest(ReadOnlySpan<byte>data){return"hello world!"u8.SequenceEqual(data);}

Codegen diff: https://www.diffchecker.com/0VOpmvMj/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side (which is likely the case for e.g. StartsWith).

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

Author:	EgorBo
Assignees:	EgorBo
Labels:	area-CodeGen-coreclr
Milestone:	-

[gfoidl]

Can be dasm for Test1 be something like

G_M000_IG01: ;; offset=0000HG_M000_IG02: ;; offset=0000H 488B01 movrax, bword ptr [rcx] 8B5108 movedx, dword ptr [rcx+08H] 83FA0B cmpedx,117404je SHORT G_M000_IG04G_M000_IG03: ;; offset=000BH 33C0 xoreax,eax EB24 jmp SHORT G_M000_IG05G_M000_IG04: ;; offset=000FH 48BA68656C6C6F20776F movrdx,0x6F77206F6C6C6568483310xorrdx, qword ptr [rax] 48B96C6F20776F726C64 movrcx,0x646C726F77206F6C48334803xorrcx, qword ptr [rax+03H] 480BD1 orrdx,rcx 0F94C0 sete al 0FB6C0 movzxrax,alG_M000_IG05: ;; offset=0033H C3 ret; Total bytes of code 52

?
So that the "constant" LHS (here "hello world"u8) is read as long-constant instead of loaded from memory. Save two memory loads.

The assembly above is produced by this simple C# approach.

Code

usingSystem.Runtime.CompilerServices;usingSystem.Runtime.InteropServices;ReadOnlySpan<byte>test="hello world"u8;Console.WriteLine(Test1(test));
#if !DEBUGfor(inti=0;i<100;++i){if(i%10==0)Thread.Sleep(100);_=Test1(test);}
#endif
staticboolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.FastSequenceEqual(data);internalstaticclassMySpanExtensions{[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolFastSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr0=Unsafe.ReadUnaligned<long>(refrightRef);longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}thrownewNotSupportedException();}}

PS: the XOR-trick here is 👍🏻

[jakobbotsch]

Are you sure this is better than the naive version for ARM64 with CCMPs? What is the ARM64 codegen diff if you create AND(EQ(IND, IND), EQ(IND, IND)) instead?

[EgorBo]

The current codegen is (comparing 16 bytes):

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]CA030021 eor x1, x1, x3F9400000 ldr x0,[x0]F9400042 ldr x2,[x2]CA020000 eor x0, x0, x2AA000020 orr x0, x1, x0F100001F cmp x0, #09A9F17E0 cset x0, eq

cmp version presumably needs ifConversion path? Because here is what I see when I follow your suggestion:

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]EB03003F cmp x1, x39A9F17E1 cset x1, eqF9400000 ldr x0,[x0]F9400042 ldr x2,[x2]EB02001F cmp x0, x29A9F17E0 cset x0, eqEA00003F tst x1, x09A9F07E0 cset x0, ne

so we need to either do this opt in codegen or earlier for that. For me arm64 codegen doesn't look too bad, it's still better than not unrolled.

[jakobbotsch]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

[EgorBo]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

still doesn't want to convert to CCMP, IsInvariantInRange check fails, presumably because of IND side effects. Still, I think the current version is better than non-unrolled

[jakobbotsch]

You should be able to insert it in the right order so that there is no interference, e.g. probably

t0 = IND
t1 = IND
t2 = IND
t3 = IND
t4 = EQ(t0, t1)
t5 = EQ(t2, t3)
t6 = AND(t4, t5)

[jakobbotsch]

Although it's a bit odd there would be interference even with

t0 =IND
t1 =IND
t2 =EQ(t0, t1)
t3 =IND
t4 =IND
t5 =EQ(t3, t4)
t6 =AND(t2, t5)

Probably something I should take a look at.

[EgorBo]

but just in case I pushed a change to move all IND nodes to the front

[gfoidl]

I have a general question: why needs this to be done in JIT and not in managed code? Is it about throughput and / or IL-size?
In some areas there's native -> managed, here it's the opposite.
This is not a rant, just a question out of curiosity. And it makes contributing a bit harder (at least for me, as I don't know the internals of JIT very much).

To implement this in pure C# something like RuntimeHelpers.IsKnownConstant(ReadOnlySpan<byte>) is missing.
Assuming such a method exists, then it could look like:

Example managed implementation

[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(/* missing piece */RuntimeHelpers.IsKnownConstant(left)){if(len>=sizeof(int)&&len<=2*sizeof(int)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);intl0=Unsafe.ReadUnaligned<int>(refleftRef);intr0=Unsafe.ReadUnaligned<int>(refrightRef);intl1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refleftRef,len-sizeof(int)));intr1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refrightRef,len-sizeof(int)));intt0=l0^r0;intt1=l1^r1;intt=t0|t1;returnt==0;}if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longr0=Unsafe.ReadUnaligned<long>(refrightRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}if(Vector128.IsHardwareAccelerated&&len>=(uint)Vector128<byte>.Count&&len<=2*(uint)Vector128<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector128<byte>l0=Vector128.LoadUnsafe(refleftRef);Vector128<byte>r0=Vector128.LoadUnsafe(refrightRef);Vector128<byte>t0=l0^r0;Vector128<byte>l1=Vector128.LoadUnsafe(refleftRef,len-(uint)Vector128<byte>.Count);Vector128<byte>r1=Vector128.LoadUnsafe(refrightRef,len-(uint)Vector128<byte>.Count);Vector128<byte>t1=l1^r1;Vector128<byte>t=t0|t1;returnt==Vector128<byte>.Zero;}if(Vector256.IsHardwareAccelerated&&len>=(uint)Vector256<byte>.Count&&len<=2*(uint)Vector256<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector256<byte>l0=Vector256.LoadUnsafe(refleftRef);Vector256<byte>r0=Vector256.LoadUnsafe(refrightRef);Vector256<byte>t0=l0^r0;Vector256<byte>l1=Vector256.LoadUnsafe(refleftRef,len-(uint)Vector256<byte>.Count);Vector256<byte>r1=Vector256.LoadUnsafe(refrightRef,len-(uint)Vector256<byte>.Count);Vector256<byte>t1=l1^r1;Vector256<byte>t=t0|t1;returnt==Vector256<byte>.Zero;}}// Current implementation of SequenceEqualCore w/o length-check (already done)returnSequenceEqualCore(left,right);}

So it's more IL and more work for the JIT to do. Are these the reasons why it's done via [Intrinsic] directly in the JIT?

[EgorBo]

@gfoidl there are two issues with the managed approach:

1. We can only detect constant length for Span in late phases of JIT, so for IsKnownConstant case we'll have to carry a large tree through all phases. And yes, it needs extra support on JIT side for IsKnownConstant
2. I tried to do the same for String.Equals unrolling once and hit two issues: first, this happy-path creates a huge amount of locals so for some deep callsites we can stop tracking locals because of that (or stop inlining more into the graph). Second, inliner's budget problem. See Unroll String.Equals for constant input [0..16] length #64821

[IDisposable]

I know this is completely unrelated to the fix or changes being done, sorry... but has anyone ever tried reversing the methodName and className tests for a performance hack in the JIT itself?

[EgorBo]

@IDisposablelookupNamedIntrinsics never show up in our JIT traces so we don't bother. This code is only executed for methods with [Intrinsic] attribute so for 99% of methods it doesn't kick in.

We could use here a Trie/binary search if it was a real problem

[gfoidl]

@EgorBo thanks for the info, I understand.

[jakobbotsch]

Seems like you could just make this function return the first new node you added, since the call was replaced anyway, and have "normal" lowering proceed from there.

[EgorBo]

Good idea, done

[EgorBo]

@EgorBo thanks for the info, I understand.

Still, filed a PR #84002 to make it possible, so now you can use IsKnownConstant(span.Length)

[jakobbotsch]

It's a bit wasteful to go looking for the uses of this given that we know the arg they come from. E.g. you could do

Suggested change

	LIR::Use lArgUse;
	LIR::Use rArgUse;
	bool lFoundUse = BlockRange().TryGetUse(lArg, &lArgUse);
	bool rFoundUse = BlockRange().TryGetUse(rArg, &rArgUse);
	assert(lFoundUse && rFoundUse);
	CallArg* lArg = call->gtArgs.GetUserArgByIndex(0);
	GenTree*& lArgNode = lArg->GetLateNode() == nullptr ? lArg->EarlyNodeRef() : lArg->LateNodeRef();
	...
	LIR::Use lArgUse(BlockRange(), &lArgNode, call);

I don't have a super strong opinion on it.

[EgorBo]

Thank, will check in a follow up once SPMI is collected - want to see if it's worth the effort to improve this expansion. jit-diff utils found around 30 methods only

[EgorBo]

Failures are #83655 and #80619

[EgorBo]

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.SequenceEqual(data);boolTest2(Span<byte>data)=>data.StartsWith("test"u8);

Codegen diff: https://www.diffchecker.com/E1laymuB/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side.
Works for StartsWith.

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}publicstaticclassPaths{publicstaticReadOnlySpan<byte>Json=>"/json"u8;publicstaticReadOnlySpan<byte>Plaintext=>"/plaintext"u8;}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

(the difference should be bigger when SIMD is enabled)

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Unroll SequenceEqual for constant length [1..15] (will add SIMD separately if this lands) for both x64 and arm64.
Example (utf8 literal):

boolTest(ReadOnlySpan<byte>data){return"hello world!"u8.SequenceEqual(data);}

Codegen diff: https://www.diffchecker.com/0VOpmvMj/

Limitations

Unfortunately, it works only when a constant span (either RVA or e.g. data.Slice(0, 10)) is on the left. It happens because we use left span's Length here:

runtime/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs

Line 1437 in 922c75c

((uint)length)*size);// If this multiplication overflows, the Span we got overflows the entire address range. There's no happy outcome for this api in such a case so we choose not to take the overhead of checking.

In theory, JIT is smart enough to perform things like:

if (x==42)
{
Foo(x); // x will be replaced with 42
}

via AssertProp, but in this case it's a bit more complicated than that. Perhaps, we can assist it with IsKnowConstant. Or we can use RHS span's length instead if we think that a constant span is more likely to appear on the right side (which is likely the case for e.g. StartsWith).

Motivation

Mainly, these comparisons in TE.

Benchmarks

[Benchmark]publicintTE_Json(){returnGetRequestType("/json"u8);}[Benchmark]publicintTE_Plaintext(){returnGetRequestType("/plaintext"u8);}[MethodImpl(MethodImplOptions.NoInlining)]privatestaticintGetRequestType(ReadOnlySpan<byte>path){// Simulate TE scenarioif(path.Length==10&&Paths.Plaintext.SequenceEqual(path)){return1;}elseif(path.Length==5&&Paths.Json.SequenceEqual(path)){return2;}return3;}staticbyte[]data1=newbyte[100];staticbyte[]data2=newbyte[100];[Benchmark]publicboolEquals_15(){returndata1.AsSpan(0,15).SequenceEqual(data2.AsSpan(0,15));}

Method	Toolchain	Mean
TE_Json	\runtime-base\corerun.exe	2.0567 ns
TE_Json	\runtime\corerun.exe	0.9143 ns
TE_Plaintext	\runtime-base\corerun.exe	1.8862 ns
TE_Plaintext	\runtime\corerun.exe	1.1548 ns
Equals_15	\runtime-base\corerun.exe	1.6169 ns
Equals_15	\runtime\corerun.exe	0.5172 ns

Author:	EgorBo
Assignees:	EgorBo
Labels:	area-CodeGen-coreclr
Milestone:	-

[gfoidl]

Can be dasm for Test1 be something like

G_M000_IG01: ;; offset=0000HG_M000_IG02: ;; offset=0000H 488B01 movrax, bword ptr [rcx] 8B5108 movedx, dword ptr [rcx+08H] 83FA0B cmpedx,117404je SHORT G_M000_IG04G_M000_IG03: ;; offset=000BH 33C0 xoreax,eax EB24 jmp SHORT G_M000_IG05G_M000_IG04: ;; offset=000FH 48BA68656C6C6F20776F movrdx,0x6F77206F6C6C6568483310xorrdx, qword ptr [rax] 48B96C6F20776F726C64 movrcx,0x646C726F77206F6C48334803xorrcx, qword ptr [rax+03H] 480BD1 orrdx,rcx 0F94C0 sete al 0FB6C0 movzxrax,alG_M000_IG05: ;; offset=0033H C3 ret; Total bytes of code 52

?
So that the "constant" LHS (here "hello world"u8) is read as long-constant instead of loaded from memory. Save two memory loads.

The assembly above is produced by this simple C# approach.

Code

usingSystem.Runtime.CompilerServices;usingSystem.Runtime.InteropServices;ReadOnlySpan<byte>test="hello world"u8;Console.WriteLine(Test1(test));
#if !DEBUGfor(inti=0;i<100;++i){if(i%10==0)Thread.Sleep(100);_=Test1(test);}
#endif
staticboolTest1(ReadOnlySpan<byte>data)=>"hello world"u8.FastSequenceEqual(data);internalstaticclassMySpanExtensions{[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolFastSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr0=Unsafe.ReadUnaligned<long>(refrightRef);longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}thrownewNotSupportedException();}}

PS: the XOR-trick here is 👍🏻

[jakobbotsch]

Are you sure this is better than the naive version for ARM64 with CCMPs? What is the ARM64 codegen diff if you create AND(EQ(IND, IND), EQ(IND, IND)) instead?

[EgorBo]

The current codegen is (comparing 16 bytes):

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]CA030021 eor x1, x1, x3F9400000 ldr x0,[x0]F9400042 ldr x2,[x2]CA020000 eor x0, x0, x2AA000020 orr x0, x1, x0F100001F cmp x0, #09A9F17E0 cset x0, eq

cmp version presumably needs ifConversion path? Because here is what I see when I follow your suggestion:

F9400001 ldr x1,[x0]F9400043 ldr x3,[x2]EB03003F cmp x1, x39A9F17E1 cset x1, eqF9400000 ldr x0,[x0]F9400042 ldr x2,[x2]EB02001F cmp x0, x29A9F17E0 cset x0, eqEA00003F tst x1, x09A9F07E0 cset x0, ne

so we need to either do this opt in codegen or earlier for that. For me arm64 codegen doesn't look too bad, it's still better than not unrolled.

[jakobbotsch]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

[EgorBo]

No, this should not need if-conversion. Are you calling lowering on these new nodes? I would expect TryLowerAndOrToCCMP to kick in and the ARM64 "naive" IR to result in ldr, ldr, ldr, ldr, cmp, ccmp, cset.

still doesn't want to convert to CCMP, IsInvariantInRange check fails, presumably because of IND side effects. Still, I think the current version is better than non-unrolled

[jakobbotsch]

You should be able to insert it in the right order so that there is no interference, e.g. probably

t0 = IND
t1 = IND
t2 = IND
t3 = IND
t4 = EQ(t0, t1)
t5 = EQ(t2, t3)
t6 = AND(t4, t5)

[jakobbotsch]

Although it's a bit odd there would be interference even with

t0 =IND
t1 =IND
t2 =EQ(t0, t1)
t3 =IND
t4 =IND
t5 =EQ(t3, t4)
t6 =AND(t2, t5)

Probably something I should take a look at.

[EgorBo]

but just in case I pushed a change to move all IND nodes to the front

[gfoidl]

I have a general question: why needs this to be done in JIT and not in managed code? Is it about throughput and / or IL-size?
In some areas there's native -> managed, here it's the opposite.
This is not a rant, just a question out of curiosity. And it makes contributing a bit harder (at least for me, as I don't know the internals of JIT very much).

To implement this in pure C# something like RuntimeHelpers.IsKnownConstant(ReadOnlySpan<byte>) is missing.
Assuming such a method exists, then it could look like:

Example managed implementation

[MethodImpl(MethodImplOptions.AggressiveInlining)]publicstaticboolSequenceEqual(thisReadOnlySpan<byte>left,ReadOnlySpan<byte>right){nuintlen=(uint)left.Length;if((uint)right.Length!=len)returnfalse;if(/* missing piece */RuntimeHelpers.IsKnownConstant(left)){if(len>=sizeof(int)&&len<=2*sizeof(int)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);intl0=Unsafe.ReadUnaligned<int>(refleftRef);intr0=Unsafe.ReadUnaligned<int>(refrightRef);intl1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refleftRef,len-sizeof(int)));intr1=Unsafe.ReadUnaligned<int>(refUnsafe.Add(refrightRef,len-sizeof(int)));intt0=l0^r0;intt1=l1^r1;intt=t0|t1;returnt==0;}if(len>=sizeof(long)&&len<=2*sizeof(long)){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);longl0=Unsafe.ReadUnaligned<long>(refleftRef);longr0=Unsafe.ReadUnaligned<long>(refrightRef);longl1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refleftRef,len-sizeof(long)));longr1=Unsafe.ReadUnaligned<long>(refUnsafe.Add(refrightRef,len-sizeof(long)));longt0=l0^r0;longt1=l1^r1;longt=t0|t1;returnt==0;}if(Vector128.IsHardwareAccelerated&&len>=(uint)Vector128<byte>.Count&&len<=2*(uint)Vector128<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector128<byte>l0=Vector128.LoadUnsafe(refleftRef);Vector128<byte>r0=Vector128.LoadUnsafe(refrightRef);Vector128<byte>t0=l0^r0;Vector128<byte>l1=Vector128.LoadUnsafe(refleftRef,len-(uint)Vector128<byte>.Count);Vector128<byte>r1=Vector128.LoadUnsafe(refrightRef,len-(uint)Vector128<byte>.Count);Vector128<byte>t1=l1^r1;Vector128<byte>t=t0|t1;returnt==Vector128<byte>.Zero;}if(Vector256.IsHardwareAccelerated&&len>=(uint)Vector256<byte>.Count&&len<=2*(uint)Vector256<byte>.Count){refbyteleftRef=refMemoryMarshal.GetReference(left);refbyterightRef=refMemoryMarshal.GetReference(right);Vector256<byte>l0=Vector256.LoadUnsafe(refleftRef);Vector256<byte>r0=Vector256.LoadUnsafe(refrightRef);Vector256<byte>t0=l0^r0;Vector256<byte>l1=Vector256.LoadUnsafe(refleftRef,len-(uint)Vector256<byte>.Count);Vector256<byte>r1=Vector256.LoadUnsafe(refrightRef,len-(uint)Vector256<byte>.Count);Vector256<byte>t1=l1^r1;Vector256<byte>t=t0|t1;returnt==Vector256<byte>.Zero;}}// Current implementation of SequenceEqualCore w/o length-check (already done)returnSequenceEqualCore(left,right);}

So it's more IL and more work for the JIT to do. Are these the reasons why it's done via [Intrinsic] directly in the JIT?

[EgorBo]

@gfoidl there are two issues with the managed approach:

1. We can only detect constant length for Span in late phases of JIT, so for IsKnownConstant case we'll have to carry a large tree through all phases. And yes, it needs extra support on JIT side for IsKnownConstant
2. I tried to do the same for String.Equals unrolling once and hit two issues: first, this happy-path creates a huge amount of locals so for some deep callsites we can stop tracking locals because of that (or stop inlining more into the graph). Second, inliner's budget problem. See Unroll String.Equals for constant input [0..16] length #64821

[IDisposable]

I know this is completely unrelated to the fix or changes being done, sorry... but has anyone ever tried reversing the methodName and className tests for a performance hack in the JIT itself?

[EgorBo]

@IDisposablelookupNamedIntrinsics never show up in our JIT traces so we don't bother. This code is only executed for methods with [Intrinsic] attribute so for 99% of methods it doesn't kick in.

We could use here a Trie/binary search if it was a real problem

[gfoidl]

@EgorBo thanks for the info, I understand.

[jakobbotsch]

Seems like you could just make this function return the first new node you added, since the call was replaced anyway, and have "normal" lowering proceed from there.

[EgorBo]

Good idea, done

[EgorBo]

@EgorBo thanks for the info, I understand.

Still, filed a PR #84002 to make it possible, so now you can use IsKnownConstant(span.Length)

[jakobbotsch]

It's a bit wasteful to go looking for the uses of this given that we know the arg they come from. E.g. you could do

Suggested change

	LIR::Use lArgUse;
	LIR::Use rArgUse;
	bool lFoundUse = BlockRange().TryGetUse(lArg, &lArgUse);
	bool rFoundUse = BlockRange().TryGetUse(rArg, &rArgUse);
	assert(lFoundUse && rFoundUse);
	CallArg* lArg = call->gtArgs.GetUserArgByIndex(0);
	GenTree*& lArgNode = lArg->GetLateNode() == nullptr ? lArg->EarlyNodeRef() : lArg->LateNodeRef();
	...
	LIR::Use lArgUse(BlockRange(), &lArgNode, call);

I don't have a super strong opinion on it.

[EgorBo]

Thank, will check in a follow up once SPMI is collected - want to see if it's worth the effort to improve this expansion. jit-diff utils found around 30 methods only

[EgorBo]

Failures are #83655 and #80619