Handle final elements in SpanHelpers.Contains for byte and char vectorized by gfoidl · Pull Request #67492 · dotnet/runtime

gfoidl · 2022-04-02T21:27:25Z

Description

Let's assume we have a searchSpace of length (n + 1) * Vector<T>.Count - k, where T is either byte or char, and k in (0, Vector<T>.Count).
So current implementation -- ignoring alignment for a moment -- can perform n vectorized operations, then falls back to sequential processing of the remaining Vector<T>.Count - k elements.

In numbers for byte, AVX2, n = 2, and k = 1:

Vector<byte>.Count = 32
length = 95
vectorized = 64
sequential = 31

So as ratio there are (Vector<T>.Count - k) / (n * Vector<T>.Count) elements that need to processed sequential.
The worst case is for k = 1 and small n, i.e. for AVX2 and k = 1 31 elements need to be processed sequential.

The proposed change avoids the sequential processing of the remaining elements by reading a final vector from the end of the searchSpace.
When exiting the standard vectorized loop, we know that the searchSpace is at least Vector<T>.Count long, so it is safe to read from that end, and the operation is idempotent too.
Thus in total we do n + 1 vectorized operations.

(Note: the same / similar approach is used in #67049, and some other places where idempotency can be used (I commented quite a few times on this 😉))

Benchmark results

Notes

JIT doesn't hoist Vector<T>.Zero outside the loop, this is done manually with this PR and that's why for length 64 (byte) and 32 (char) a speedup is shown
for length 95 (byte) and 47 (char) the described effect is most visible, as this is the worst case, meaning most elements were processed sequential before this PR
by not jumping back to the sequential path quite a lot of comparisons get saved too, resulting in a more streamlined and predicatable instruction-flow

For the benchmarks the searchSpace is aligned to 32 bytes, to have reproducable results.

machine info

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19043.1586 (21H1/May2021Update)
Intel Core i7-7700HQ CPU 2.80GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=7.0.100-preview.4.22181.7
[Host] : .NET 7.0.0 (7.0.22.17907), X64 RyuJIT
DefaultJob : .NET 7.0.0 (7.0.22.17907), X64 RyuJIT

bool Contains(ref byte searchSpace, byte value, int length)

| Method | Length | Mean | Error | StdDev | Ratio |
|-------- |------- |----------:|----------:|----------:|------:|
| Default | 63 | 14.623 ns | 0.3486 ns | 0.9946 ns | 1.00 |
| PR | 63 | 14.479 ns | 0.2283 ns | 0.2135 ns | 0.99 |
| | | | | | |
| Default | 64 | 4.419 ns | 0.1277 ns | 0.1790 ns | 1.00 |
| PR | 64 | 3.963 ns | 0.0494 ns | 0.0462 ns | 0.87 |
| | | | | | |
| Default | 65 | 6.412 ns | 0.1566 ns | 0.1608 ns | 1.00 |
| PR | 65 | 4.469 ns | 0.0229 ns | 0.0203 ns | 0.70 |
| | | | | | |
| Default | 95 | 11.318 ns | 0.1033 ns | 0.0966 ns | 1.00 |
| PR | 95 | 4.502 ns | 0.0543 ns | 0.0508 ns | 0.40 |
| | | | | | |
| Default | 100 | 8.343 ns | 0.1312 ns | 0.1096 ns | 1.00 |
| PR | 100 | 5.246 ns | 0.1376 ns | 0.1287 ns | 0.63 |

bool Contains(ref char searchSpace, char value, int length)

| Method | Length | Mean | Error | StdDev | Ratio |
|-------- |------- |----------:|----------:|----------:|------:|
| Default | 31 | 10.349 ns | 0.2485 ns | 0.6325 ns | 1.00 |
| PR | 31 | 10.232 ns | 0.2426 ns | 0.5998 ns | 0.99 |
| | | | | | |
| Default | 32 | 8.498 ns | 0.1741 ns | 0.1454 ns | 1.00 |
| PR | 32 | 6.177 ns | 0.1473 ns | 0.1967 ns | 0.71 |
| | | | | | |
| Default | 33 | 8.751 ns | 0.2081 ns | 0.1946 ns | 1.00 |
| PR | 33 | 4.655 ns | 0.1279 ns | 0.1313 ns | 0.53 |
| | | | | | |
| Default | 47 | 9.665 ns | 0.2301 ns | 0.3373 ns | 1.00 |
| PR | 47 | 4.563 ns | 0.0421 ns | 0.0374 ns | 0.46 |
| | | | | | |
| Default | 100 | 9.096 ns | 0.1002 ns | 0.0937 ns | 1.00 |
| PR | 100 | 8.182 ns | 0.1001 ns | 0.0836 ns | 0.90 |

Machine code (x64)

SpanHelpers.Contains(byte)

; SpanHelpersContainsByteBenchmark.Default()movrdx,[rcx+8]movzxeax,byte ptr [rcx+14]movr8d,[rcx+10]movrcx,rdxmovedx,eaxjmp qword ptr [7FFB35F51420]; Total bytes of code 23; SpanHelpersContainsByteBenchmark.Contains(Byte ByRef, Byte, Int32)vzerouppermovzxeax,dlmovedx,eaxxorr9d,r9dmovr10d,r8dmovr11,r10cmpr8d,40jl short M01_L00movr11,rcxandr11,1Fnegr11addr11,20andr11,1FM01_L00:cmpr11,8jb near ptr M01_L02M01_L01:addr11,0FFFFFFFFFFFFFFF8movzxr8d,byte ptr [rcx+r9]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+1]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+2]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+3]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+4]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+5]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+6]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+7]cmpr8d,edxje near ptr M01_L09addr9,8cmpr11,8jae near ptr M01_L01M01_L02:cmpr11,4jb short M01_L03addr11,0FFFFFFFFFFFFFFFCmovzxr8d,byte ptr [rcx+r9]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+1]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+2]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+3]cmpr8d,edxje near ptr M01_L09addr9,4M01_L03:testr11,r11je short M01_L05M01_L04:decr11movzxr8d,byte ptr [rcx+r9]cmpr8d,edxje short M01_L09incr9testr11,r11jne short M01_L04M01_L05:cmpr9,r10jae short M01_L08movr11,r10subr11,r9andr11,0FFFFFFFFFFFFFFE0imulr8d,eax,1010101 vmovd xmm0,r8dvpbroadcastdymm0,xmm0cmpr11,r9jbe short M01_L07nop dword ptr [rax]nop dword ptr [rax+rax]M01_L06: vpcmpeqb ymm1,ymm0,[rcx+r9] vxorps ymm2,ymm2,ymm2vpxorymm1,ymm2,ymm1vptestymm1,ymm1jne short M01_L09addr9,20cmpr11,r9ja short M01_L06M01_L07:cmpr9,r10jae short M01_L08movr11,r10subr11,r9jmp near ptr M01_L00M01_L08:xoreax,eaxvzeroupperretM01_L09:moveax,1vzeroupperret; Total bytes of code 397

; SpanHelpersContainsByteBenchmark.PR()movrdx,[rcx+8]movzxeax,byte ptr [rcx+14]movr8d,[rcx+10]movrcx,rdxmovedx,eaxjmp qword ptr [7FFB35F61438]; Total bytes of code 23; SpanHelpersContainsByteBenchmark.Contains_PR(Byte ByRef, Byte, Int32)pushrdipushrsivzerouppermovzxeax,dlmovedx,eaxxorr9d,r9dmovr10d,r8dmovr11,r10cmpr8d,40jl short M01_L00movr11,rcxandr11,1Fnegr11addr11,20andr11,1FM01_L00:cmpr11,8jb short M01_L02nop dword ptr [rax]M01_L01:addr11,0FFFFFFFFFFFFFFF8learsi,[rcx+r9]movzxedi,byte ptr [rsi]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+1]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+2]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+3]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+4]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+5]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+6]cmpedx,edije near ptr M01_L09movzxesi,byte ptr [rsi+7]cmpedx,esije near ptr M01_L09addr9,8cmpr11,8jae short M01_L01M01_L02:cmpr11,4jb short M01_L03addr11,0FFFFFFFFFFFFFFFClearsi,[rcx+r9]movzxedi,byte ptr [rsi]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+1]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+2]cmpedx,edije near ptr M01_L09movzxesi,byte ptr [rsi+3]cmpedx,esije near ptr M01_L09addr9,4M01_L03:testr11,r11je short M01_L05nop dword ptr [rax+rax]M01_L04:decr11movzxesi,byte ptr [rcx+r9]cmpesi,edxje short M01_L09incr9testr11,r11jne short M01_L04M01_L05:cmpr9,r10jae short M01_L08movr11,r10subr11,r9andr11,0FFFFFFFFFFFFFFE0 vxorps ymm0,ymm0,ymm0imuleax,1010101 vmovd xmm1,eaxvpbroadcastdymm1,xmm1cmpr9,r11jae short M01_L07M01_L06: vpcmpeqb ymm2,ymm1,[rcx+r9]vpxorymm2,ymm0,ymm2vptestymm2,ymm2jne short M01_L09addr9,20cmpr9,r11jb short M01_L06M01_L07:cmpr9,r10jae short M01_L08addr8d,0FFFFFFE0movr9d,r8d vpcmpeqb ymm2,ymm1,[rcx+r9]vpxorymm0,ymm0,ymm2vptestymm0,ymm0jne short M01_L09M01_L08:xoreax,eaxvzeroupperpoprsipoprdiretM01_L09:moveax,1vzeroupperpoprsipoprdiret; Total bytes of code 389

SpanHelpers.Contains(char)

; SpanHelpersContainsCharBenchmark.Default()movrdx,[rcx+8]movzxeax,word ptr [rcx+14]movr8d,[rcx+10]movrcx,rdxmovedx,eaxjmp qword ptr [7FFB35F51420]; Total bytes of code 23; SpanHelpersContainsCharBenchmark.Contains(Char ByRef, Char, Int32)pushraxvzeroupperxoreax,eaxmov[rsp],raxmov[rsp],rcxmovsxdrax,r8dlear9,[rcx+rax*2]cmpr8d,20jl short M01_L00movr8d,ecxandr8d,1Fmoveax,r8dshreax,1Faddeax,r8dsareax,1movr8d,eaxnegr8daddr8d,10andr8d,0FM01_L00:cmpr8d,4jl short M01_L02movzxr10d,dxM01_L01:addr8d,0FFFFFFFCmovzxeax,word ptr [rcx]cmpr10d,eaxje near ptr M01_L08movzxeax,word ptr [rcx+2]cmpr10d,eaxje near ptr M01_L08movzxeax,word ptr [rcx+4]cmpr10d,eaxje near ptr M01_L08movzxeax,word ptr [rcx+6]cmpr10d,eaxje near ptr M01_L08addrcx,8cmpr8d,4jge short M01_L01M01_L02:testr8d,r8djle short M01_L04movzxr10d,dxnopM01_L03:decr8dmovzxeax,word ptr [rcx]cmpr10d,eaxje near ptr M01_L08addrcx,2testr8d,r8djg short M01_L03M01_L04:cmprcx,r9jae short M01_L07movr8,r9subr8,rcxmovrax,r8shrrax,3Faddrax,r8sarrax,1movr8d,eaxandr8d,0FFFFFFF0movzxr10d,dximuleax,r10d,10001 vmovd xmm0,eaxvpbroadcastdymm0,xmm0testr8d,r8djle short M01_L06M01_L05: vpcmpeqw ymm1,ymm0,[rcx] vxorps ymm2,ymm2,ymm2vpxorymm1,ymm2,ymm1vptestymm1,ymm1jne short M01_L08addrcx,20addr8d,0FFFFFFF0testr8d,r8djg short M01_L05M01_L06:cmprcx,r9jae short M01_L07movr8,r9subr8,rcxmovrax,r8shrrax,3Faddrax,r8sarrax,1movr8d,eaxjmp near ptr M01_L00M01_L07:xoreax,eaxvzeroupperaddrsp,8retM01_L08:moveax,1vzeroupperaddrsp,8ret; Total bytes of code 311

; SpanHelpersContainsCharBenchmark.PR()movrdx,[rcx+8]movzxeax,word ptr [rcx+14]movr8d,[rcx+10]movrcx,rdxmovedx,eaxjmp qword ptr [7FFB35F51438]; Total bytes of code 23; SpanHelpersContainsCharBenchmark.Contains_PR(Char ByRef, Char, Int32)pushrsisubrsp,10vzeroupperxoreax,eaxmov[rsp+8],raxmov[rsp+8],rcxxorr9d,r9dmovr10d,r8dmovr11,r10cmpr8d,20jl short M01_L00movr11d,ecxandr11d,1Fshrr11d,1moveax,r11dnegeaxaddeax,10andeax,0Fmovr11d,eaxM01_L00:cmpr11,4jb short M01_L02movzxr8d,dxM01_L01:addr11,0FFFFFFFFFFFFFFFClearax,[rcx+r9*2]movzxesi,word ptr [rax]cmpr8d,esije near ptr M01_L08movzxesi,word ptr [rax+2]cmpr8d,esije near ptr M01_L08movzxesi,word ptr [rax+4]cmpr8d,esije near ptr M01_L08movzxeax,word ptr [rax+6]cmpr8d,eaxje near ptr M01_L08addr9,4cmpr11,4jae short M01_L01M01_L02:testr11,r11je short M01_L04movzxr8d,dxnop dword ptr [rax+rax]nop dword ptr [rax+rax]M01_L03:decr11movzxeax,word ptr [rcx+r9*2]cmpeax,r8dje near ptr M01_L08incr9testr11,r11jne short M01_L03M01_L04:cmpr9,r10jae short M01_L07movr11,r10subr11,r9andr11,0FFFFFFFFFFFFFFF0 vxorps ymm0,ymm0,ymm0movzxr8d,dximuleax,r8d,10001 vmovd xmm1,eaxvpbroadcastdymm1,xmm1cmpr9,r11jae short M01_L06nop word ptr [rax+rax]M01_L05: vpcmpeqw ymm2,ymm1,[rcx+r9*2]vpxorymm2,ymm0,ymm2vptestymm2,ymm2jne short M01_L08addr9,10cmpr9,r11jb short M01_L05M01_L06:cmpr9,r10jae short M01_L07 vpcmpeqw ymm2,ymm1,[rcx+r10*2+0FFE0]vpxorymm0,ymm0,ymm2vptestymm0,ymm0jne short M01_L08M01_L07:xoreax,eaxvzeroupperaddrsp,10poprsiretM01_L08:moveax,1vzeroupperaddrsp,10poprsiret; Total bytes of code 314

👉 If this looks good, I'd like to look into IndexOf, etc. too.

…ctorized

ghost · 2022-04-02T21:27:36Z

Tagging subscribers to this area: @dotnet/area-system-memory
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

Let's assume we have a searchSpace of length (n + 1) * Vector<T>.Count - k, where T is either byte or char, and k in (0, Vector<T>.Count).
So current implementation -- ignoring alignment for a moment -- can perform n vectorized operations, then falls back to sequential processing of the remaining Vector<T>.Count - k elements.

In numbers for byte, AVX2, n = 2, and k = 1:

Vector<byte>.Count = 32
length = 95
vectorized = 64
sequential = 31

So as ratio there are (Vector<T>.Count - k) / (n * Vector<T>.Count) elements that need to processed sequential.
The worst case is for k = 1 and small n, i.e. for AVX2 and k = 1 31 elements need to be processed sequential.

The proposed change avoids the sequential processing of the remaining elements by reading a final vector from the end of the searchSpace.
When exiting the standard vectorized loop, we know that the searchSpace is at least Vector<T>.Count long, so it is safe to read from that end, and the operation is idempotent too.
Thus in total we do n + 1 vectorized operations.

(Note: the same / similar approach is used in #67049, and some other places where idempotency can be used (I commented quite a few times on this 😉))

Benchmark results

Notes

JIT doesn't hoist Vector<T>.Zero outside the loop, this is done manually with this PR and that's why for length 64 (byte) and 32 (char) a speedup is shown
for length 95 (byte) and 47 (char) the described effect is most visible, as this is the worst case, meaning most elements were processed sequential before this PR
by not jumping back to the sequential path quite a lot of comparisons get saved too, resulting in a more streamlined and predicatable instruction-flow

For the benchmarks the searchSpace is aligned to 32 bytes, to have reproducable results.

machine info

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19043.1586 (21H1/May2021Update)
Intel Core i7-7700HQ CPU 2.80GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=7.0.100-preview.4.22181.7
[Host] : .NET 7.0.0 (7.0.22.17907), X64 RyuJIT
DefaultJob : .NET 7.0.0 (7.0.22.17907), X64 RyuJIT

bool Contains(ref byte searchSpace, byte value, int length)

| Method | Length | Mean | Error | StdDev | Ratio |
|-------- |------- |----------:|----------:|----------:|------:|
| Default | 63 | 14.623 ns | 0.3486 ns | 0.9946 ns | 1.00 |
| PR | 63 | 14.479 ns | 0.2283 ns | 0.2135 ns | 0.99 |
| | | | | | |
| Default | 64 | 4.419 ns | 0.1277 ns | 0.1790 ns | 1.00 |
| PR | 64 | 3.963 ns | 0.0494 ns | 0.0462 ns | 0.87 |
| | | | | | |
| Default | 65 | 6.412 ns | 0.1566 ns | 0.1608 ns | 1.00 |
| PR | 65 | 4.469 ns | 0.0229 ns | 0.0203 ns | 0.70 |
| | | | | | |
| Default | 95 | 11.318 ns | 0.1033 ns | 0.0966 ns | 1.00 |
| PR | 95 | 4.502 ns | 0.0543 ns | 0.0508 ns | 0.40 |
| | | | | | |
| Default | 100 | 8.343 ns | 0.1312 ns | 0.1096 ns | 1.00 |
| PR | 100 | 5.246 ns | 0.1376 ns | 0.1287 ns | 0.63 |

bool Contains(ref char searchSpace, char value, int length)

| Method | Length | Mean | Error | StdDev | Ratio |
|-------- |------- |----------:|----------:|----------:|------:|
| Default | 31 | 10.349 ns | 0.2485 ns | 0.6325 ns | 1.00 |
| PR | 31 | 10.232 ns | 0.2426 ns | 0.5998 ns | 0.99 |
| | | | | | |
| Default | 32 | 8.498 ns | 0.1741 ns | 0.1454 ns | 1.00 |
| PR | 32 | 6.177 ns | 0.1473 ns | 0.1967 ns | 0.71 |
| | | | | | |
| Default | 33 | 8.751 ns | 0.2081 ns | 0.1946 ns | 1.00 |
| PR | 33 | 4.655 ns | 0.1279 ns | 0.1313 ns | 0.53 |
| | | | | | |
| Default | 47 | 9.665 ns | 0.2301 ns | 0.3373 ns | 1.00 |
| PR | 47 | 4.563 ns | 0.0421 ns | 0.0374 ns | 0.46 |
| | | | | | |
| Default | 100 | 9.096 ns | 0.1002 ns | 0.0937 ns | 1.00 |
| PR | 100 | 8.182 ns | 0.1001 ns | 0.0836 ns | 0.90 |

Machine code (x64)

SpanHelpers.Contains(byte)

; SpanHelpersContainsByteBenchmark.Default()movrdx,[rcx+8]movzxeax,byte ptr [rcx+14]movr8d,[rcx+10]movrcx,rdxmovedx,eaxjmp qword ptr [7FFB35F51420]; Total bytes of code 23; SpanHelpersContainsByteBenchmark.Contains(Byte ByRef, Byte, Int32)vzerouppermovzxeax,dlmovedx,eaxxorr9d,r9dmovr10d,r8dmovr11,r10cmpr8d,40jl short M01_L00movr11,rcxandr11,1Fnegr11addr11,20andr11,1FM01_L00:cmpr11,8jb near ptr M01_L02M01_L01:addr11,0FFFFFFFFFFFFFFF8movzxr8d,byte ptr [rcx+r9]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+1]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+2]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+3]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+4]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+5]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+6]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+7]cmpr8d,edxje near ptr M01_L09addr9,8cmpr11,8jae near ptr M01_L01M01_L02:cmpr11,4jb short M01_L03addr11,0FFFFFFFFFFFFFFFCmovzxr8d,byte ptr [rcx+r9]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+1]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+2]cmpr8d,edxje near ptr M01_L09movzxr8d,byte ptr [rcx+r9+3]cmpr8d,edxje near ptr M01_L09addr9,4M01_L03:testr11,r11je short M01_L05M01_L04:decr11movzxr8d,byte ptr [rcx+r9]cmpr8d,edxje short M01_L09incr9testr11,r11jne short M01_L04M01_L05:cmpr9,r10jae short M01_L08movr11,r10subr11,r9andr11,0FFFFFFFFFFFFFFE0imulr8d,eax,1010101 vmovd xmm0,r8dvpbroadcastdymm0,xmm0cmpr11,r9jbe short M01_L07nop dword ptr [rax]nop dword ptr [rax+rax]M01_L06: vpcmpeqb ymm1,ymm0,[rcx+r9] vxorps ymm2,ymm2,ymm2vpxorymm1,ymm2,ymm1vptestymm1,ymm1jne short M01_L09addr9,20cmpr11,r9ja short M01_L06M01_L07:cmpr9,r10jae short M01_L08movr11,r10subr11,r9jmp near ptr M01_L00M01_L08:xoreax,eaxvzeroupperretM01_L09:moveax,1vzeroupperret; Total bytes of code 397

; SpanHelpersContainsByteBenchmark.PR()movrdx,[rcx+8]movzxeax,byte ptr [rcx+14]movr8d,[rcx+10]movrcx,rdxmovedx,eaxjmp qword ptr [7FFB35F61438]; Total bytes of code 23; SpanHelpersContainsByteBenchmark.Contains_PR(Byte ByRef, Byte, Int32)pushrdipushrsivzerouppermovzxeax,dlmovedx,eaxxorr9d,r9dmovr10d,r8dmovr11,r10cmpr8d,40jl short M01_L00movr11,rcxandr11,1Fnegr11addr11,20andr11,1FM01_L00:cmpr11,8jb short M01_L02nop dword ptr [rax]M01_L01:addr11,0FFFFFFFFFFFFFFF8learsi,[rcx+r9]movzxedi,byte ptr [rsi]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+1]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+2]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+3]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+4]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+5]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+6]cmpedx,edije near ptr M01_L09movzxesi,byte ptr [rsi+7]cmpedx,esije near ptr M01_L09addr9,8cmpr11,8jae short M01_L01M01_L02:cmpr11,4jb short M01_L03addr11,0FFFFFFFFFFFFFFFClearsi,[rcx+r9]movzxedi,byte ptr [rsi]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+1]cmpedx,edije near ptr M01_L09movzxedi,byte ptr [rsi+2]cmpedx,edije near ptr M01_L09movzxesi,byte ptr [rsi+3]cmpedx,esije near ptr M01_L09addr9,4M01_L03:testr11,r11je short M01_L05nop dword ptr [rax+rax]M01_L04:decr11movzxesi,byte ptr [rcx+r9]cmpesi,edxje short M01_L09incr9testr11,r11jne short M01_L04M01_L05:cmpr9,r10jae short M01_L08movr11,r10subr11,r9andr11,0FFFFFFFFFFFFFFE0 vxorps ymm0,ymm0,ymm0imuleax,1010101 vmovd xmm1,eaxvpbroadcastdymm1,xmm1cmpr9,r11jae short M01_L07M01_L06: vpcmpeqb ymm2,ymm1,[rcx+r9]vpxorymm2,ymm0,ymm2vptestymm2,ymm2jne short M01_L09addr9,20cmpr9,r11jb short M01_L06M01_L07:cmpr9,r10jae short M01_L08addr8d,0FFFFFFE0movr9d,r8d vpcmpeqb ymm2,ymm1,[rcx+r9]vpxorymm0,ymm0,ymm2vptestymm0,ymm0jne short M01_L09M01_L08:xoreax,eaxvzeroupperpoprsipoprdiretM01_L09:moveax,1vzeroupperpoprsipoprdiret; Total bytes of code 389

SpanHelpers.Contains(char)

; SpanHelpersContainsCharBenchmark.Default()movrdx,[rcx+8]movzxeax,word ptr [rcx+14]movr8d,[rcx+10]movrcx,rdxmovedx,eaxjmp qword ptr [7FFB35F51420]; Total bytes of code 23; SpanHelpersContainsCharBenchmark.Contains(Char ByRef, Char, Int32)pushraxvzeroupperxoreax,eaxmov[rsp],raxmov[rsp],rcxmovsxdrax,r8dlear9,[rcx+rax*2]cmpr8d,20jl short M01_L00movr8d,ecxandr8d,1Fmoveax,r8dshreax,1Faddeax,r8dsareax,1movr8d,eaxnegr8daddr8d,10andr8d,0FM01_L00:cmpr8d,4jl short M01_L02movzxr10d,dxM01_L01:addr8d,0FFFFFFFCmovzxeax,word ptr [rcx]cmpr10d,eaxje near ptr M01_L08movzxeax,word ptr [rcx+2]cmpr10d,eaxje near ptr M01_L08movzxeax,word ptr [rcx+4]cmpr10d,eaxje near ptr M01_L08movzxeax,word ptr [rcx+6]cmpr10d,eaxje near ptr M01_L08addrcx,8cmpr8d,4jge short M01_L01M01_L02:testr8d,r8djle short M01_L04movzxr10d,dxnopM01_L03:decr8dmovzxeax,word ptr [rcx]cmpr10d,eaxje near ptr M01_L08addrcx,2testr8d,r8djg short M01_L03M01_L04:cmprcx,r9jae short M01_L07movr8,r9subr8,rcxmovrax,r8shrrax,3Faddrax,r8sarrax,1movr8d,eaxandr8d,0FFFFFFF0movzxr10d,dximuleax,r10d,10001 vmovd xmm0,eaxvpbroadcastdymm0,xmm0testr8d,r8djle short M01_L06M01_L05: vpcmpeqw ymm1,ymm0,[rcx] vxorps ymm2,ymm2,ymm2vpxorymm1,ymm2,ymm1vptestymm1,ymm1jne short M01_L08addrcx,20addr8d,0FFFFFFF0testr8d,r8djg short M01_L05M01_L06:cmprcx,r9jae short M01_L07movr8,r9subr8,rcxmovrax,r8shrrax,3Faddrax,r8sarrax,1movr8d,eaxjmp near ptr M01_L00M01_L07:xoreax,eaxvzeroupperaddrsp,8retM01_L08:moveax,1vzeroupperaddrsp,8ret; Total bytes of code 311

; SpanHelpersContainsCharBenchmark.PR()movrdx,[rcx+8]movzxeax,word ptr [rcx+14]movr8d,[rcx+10]movrcx,rdxmovedx,eaxjmp qword ptr [7FFB35F51438]; Total bytes of code 23; SpanHelpersContainsCharBenchmark.Contains_PR(Char ByRef, Char, Int32)pushrsisubrsp,10vzeroupperxoreax,eaxmov[rsp+8],raxmov[rsp+8],rcxxorr9d,r9dmovr10d,r8dmovr11,r10cmpr8d,20jl short M01_L00movr11d,ecxandr11d,1Fshrr11d,1moveax,r11dnegeaxaddeax,10andeax,0Fmovr11d,eaxM01_L00:cmpr11,4jb short M01_L02movzxr8d,dxM01_L01:addr11,0FFFFFFFFFFFFFFFClearax,[rcx+r9*2]movzxesi,word ptr [rax]cmpr8d,esije near ptr M01_L08movzxesi,word ptr [rax+2]cmpr8d,esije near ptr M01_L08movzxesi,word ptr [rax+4]cmpr8d,esije near ptr M01_L08movzxeax,word ptr [rax+6]cmpr8d,eaxje near ptr M01_L08addr9,4cmpr11,4jae short M01_L01M01_L02:testr11,r11je short M01_L04movzxr8d,dxnop dword ptr [rax+rax]nop dword ptr [rax+rax]M01_L03:decr11movzxeax,word ptr [rcx+r9*2]cmpeax,r8dje near ptr M01_L08incr9testr11,r11jne short M01_L03M01_L04:cmpr9,r10jae short M01_L07movr11,r10subr11,r9andr11,0FFFFFFFFFFFFFFF0 vxorps ymm0,ymm0,ymm0movzxr8d,dximuleax,r8d,10001 vmovd xmm1,eaxvpbroadcastdymm1,xmm1cmpr9,r11jae short M01_L06nop word ptr [rax+rax]M01_L05: vpcmpeqw ymm2,ymm1,[rcx+r9*2]vpxorymm2,ymm0,ymm2vptestymm2,ymm2jne short M01_L08addr9,10cmpr9,r11jb short M01_L05M01_L06:cmpr9,r10jae short M01_L07 vpcmpeqw ymm2,ymm1,[rcx+r10*2+0FFE0]vpxorymm0,ymm0,ymm2vptestymm0,ymm0jne short M01_L08M01_L07:xoreax,eaxvzeroupperaddrsp,10poprsiretM01_L08:moveax,1vzeroupperaddrsp,10poprsiret; Total bytes of code 314

👉 If this looks good, I'd like to look into IndexOf, etc. too.

Author:	gfoidl
Assignees:	-
Labels:	`area-System.Memory`, `community-contribution`
Milestone:	-

danmoseley · 2022-04-03T01:58:55Z

It seems we're missing benchmarks for this? (If so can we add yours?)

VS didn't do this for comments (at least in my setup) automatically :-(

gfoidl · 2022-04-03T11:28:11Z

It seems we're missing benchmarks for this? (If so can we add yours?)

Sure 😃 dotnet/performance#2347

EgorBo · 2022-04-03T13:02:39Z

-var matches = Vector.Equals(values, LoadVector(ref searchSpace, offset));
- if (Vector<byte>.Zero.Equals(matches))
+ matches = Vector.Equals(values, LoadVector(ref searchSpace, offset));
+ if (zero.Equals(matches))


looks like there is CQ issue in this pattern:
(feel free to file an issue)

More over you don't need to hoist it - it should not be used

Ah, vec1 == vec2 emits better code.
For the char-overload:
M01_L05: vpcmpeqw ymm2,ymm1,[rcx+r9*2] - vpxor ymm2,ymm0,ymm2 vptest ymm2,ymm2 jne short M01_L08 add r9,10 cmp r9,r11 jb short M01_L05 M01_L06: cmp r9,r10 jae short M01_L07 vpcmpeqw ymm2,ymm1,[rcx+r10*2+0FFE0] - vpxor ymm0,ymm0,ymm2 vptest ymm0,ymm0 jne short M01_L08
Thanks for the hint!
Will create an issue for that --> #67500

… issue Cf. dotnet#67492 (comment)

adamsitnik

LGTM, thank you @gfoidl !

adamsitnik · 2022-04-25T14:37:13Z

 lengthToExamine = UnalignedCountVector(ref searchSpace);
 }

- SequentialScan:


a removal of anything goto-related is always welcomed 👍

adamsitnik · 2022-04-25T14:43:50Z

 }

- if (offset < (nuint)(uint)length)
+ // The total length is at least Vector<byte>.Count, so instead of falling back to a


thank you for adding the comment (otherwise it would not be obvious to me) 👍

AndyAyersMS · 2022-04-27T17:05:21Z

Did we expect any perf regressions from this? Seems like it might be related to dotnet/perf-autofiling-issues#4884

.

gfoidl · 2022-04-30T10:56:50Z

Did we expect any perf regressions from this?

No regression is expected, rather it should be an improvement.

When I check the benchmark-codeSpanHelpers.Contains isn't hit?

What is a proper way to investigate this regression?

gfoidl added 2 commits April 2, 2022 22:41

Handle final elements in SpanHelpers.Contains(ref byte, byte, int) ve…
89cc087
…ctorized

Handle final elements in SpanHelpers.Contains(ref char, char, int) ve…
4763c44
…ctorized

ghost added community-contribution Indicates that the PR has been added by a community member area-System.Memory labels Apr 2, 2022

This comment was marked as outdated.
Sign in to view

Fixed error SA1028: Code should not contain trailing whitespace
567e692
VS didn't do this for comments (at least in my setup) automatically :-(

gfoidl mentioned this pull request Apr 3, 2022
Added Benchmark for SpanHelpers.Contains {byte, char} dotnet/performance#2347
Open

EgorBo reviewed Apr 3, 2022
View reviewed changes

Comment threadsrc/libraries/System.Private.CoreLib/src/System/SpanHelpers.Char.cs

EgorBo reviewed Apr 3, 2022
View reviewed changes

EgorBo approved these changes Apr 3, 2022
View reviewed changes

Use equality operator instead of Vector<T>.Zero.Equals due to codegen…
debc817
… issue Cf. dotnet#67492 (comment)

gfoidl mentioned this pull request Apr 3, 2022
Vector<T>.Zero.Equals(vec) produces less ideal code than vec == Vector<T>.Zero #67500
Closed

adamsitnik self-assigned this Apr 25, 2022

adamsitnik added the tenet-performance Performance related issue label Apr 25, 2022

adamsitnik approved these changes Apr 25, 2022
View reviewed changes

adamsitnik merged commit 1958c7e into dotnet:mainApr 25, 2022

gfoidl deleted the spanhelpers_final_elements_opt branch April 25, 2022 15:19

AndyAyersMS mentioned this pull request Apr 27, 2022
[Perf] Changes at 4/25/2022 3:15:54 PM dotnet/perf-autofiling-issues#4884
Closed

ghost locked as resolved and limited conversation to collaborators May 30, 2022

Uh oh!

Conversation

gfoidl commented Apr 2, 2022

Description

Benchmark results

Notes

bool Contains(ref byte searchSpace, byte value, int length)

bool Contains(ref char searchSpace, char value, int length)

Machine code (x64)

Uh oh!

ghost commented Apr 2, 2022

Description

Benchmark results

Notes

bool Contains(ref byte searchSpace, byte value, int length)

bool Contains(ref char searchSpace, char value, int length)

Machine code (x64)

Uh oh!

This comment was marked as outdated.

danmoseley commented Apr 3, 2022

Uh oh!

gfoidl commented Apr 3, 2022

Uh oh!

Uh oh!

EgorBoApr 3, 2022

Choose a reason for hiding this comment

Uh oh!

EgorBoApr 3, 2022

Choose a reason for hiding this comment

Uh oh!

gfoidlApr 3, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

adamsitnikApr 25, 2022

Choose a reason for hiding this comment

Uh oh!

adamsitnikApr 25, 2022

Choose a reason for hiding this comment

Uh oh!

AndyAyersMS commented Apr 27, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gfoidl commented Apr 30, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gfoidlApr 3, 2022•
edited
Loading

AndyAyersMS commented Apr 27, 2022•
edited
Loading