JIT: Unblock Vector###<long> intrinsics on x86

[saucecontrol]

Resolves#11626

This resolves a large number of TODOs around HWIntrinsic expansion involving scalar longs on x86.

The most significant change here is in promoting CreateScalar and ToScalar to be code generating intrinsics instead of converting them to other intrinsics at lowering. This was necessary in order to handle emitting movq for scalar long loads/stores but also unlocks several other optimizations since we can now allow CreateScalar and ToScalar to be contained and can specialize codegen depending on whether they end up loading/storing from/to memory or not. Some example improvements on x64:

Vector128.CreateScalar(ref float):

- vinsertps xmm0, xmm0, dword ptr [rbp+0x10], 14+ vmovss xmm0, dword ptr [rbp+0x10]

Vector128.CreateScalar(ref double):

- vxorps xmm0, xmm0, xmm0- vmovsd xmm1, qword ptr [rbp-0x08]- vmovsd xmm0, xmm0, xmm1+ vmovsd xmm0, qword ptr [rbp-0x08]

ref byte = Vector128<byte>.ToScalar():

- vmovd r9d, xmm3- mov byte ptr [r10], r9b+ vpextrb byte ptr [r10], xmm3, 0

Vector<byte>.ToScalar()

- vmovups ymm0, ymmword ptr [esp+0x04]- vmovd eax, xmm0- movzx eax, al+ movzx eax, byte ptr [esp+0x04]

And the less realistic, but still interesting
Sse.AddScalar(Vector128.CreateScalar(ref float), Vector128.CreateScalar(ref float)).ToScalar():

- xorps xmm0, xmm0- movss xmm1, dword ptr [rcx]- movss xmm0, xmm1- xorps xmm1, xmm1- movss xmm2, dword ptr [rdx]- movss xmm1, xmm2- addss xmm0, xmm1+ movss xmm0, dword ptr [rcx]+ addss xmm0, dword ptr [rdx]

This also removes some redundant casts for CreateScalar of small types. Previously, a zero-extending cast was inserted unconditionally and was sometimes removed by peephole opt on x64 but often wasn't.

Vector128.CreateScalar(short):

- movsx rax, dx- movzx rax, ax- movd xmm0, rax+ movzx rax, dx+ movd xmm0, eax

Vector128.CreateScalar(checked((byte)val)):

 cmp edx, 255
ja SHORT G_M000_IG04
mov eax, edx
- movzx rax, al- vmovd xmm0, rax+ vmovd xmm0, eax

Vector128.CreateScalar(ref sbyte):

- movsx rax, byte ptr [rdx]- movzx rax, al- vmovd xmm0, rax+ movzx rax, byte ptr [rdx]+ vmovd xmm0, eax

x86 diffs are much more significant, because of the newly-enabled intrinsic expansion:

Collection	Base size (bytes)	Diff size (bytes)	PerfScore in Diffs
benchmarks.run.windows.x86.checked.mch	7,149,204	-1,892	-2.17%
benchmarks.run_pgo.windows.x86.checked.mch	46,986,713	-738	+0.03%
benchmarks.run_tiered.windows.x86.checked.mch	9,470,045	-976	+0.11%
coreclr_tests.run.windows.x86.checked.mch	320,065,247	-205,564	-6.41%
libraries.crossgen2.windows.x86.checked.mch	31,314,339	-15,854	-4.11%
libraries.pmi.windows.x86.checked.mch	34,326,245	-14,416	-2.19%
libraries_tests.run.windows.x86.Release.mch	215,517,600	-55,366	-2.41%
libraries_tests_no_tiered_compilation.run.windows.x86.Release.mch	115,783,488	-80,576	-3.65%
realworld.run.windows.x86.checked.mch	9,587,950	-467	-0.45%

[saucecontrol]

This is ready for review.
cc @tannergooding

[saucecontrol]

This condition, added in #72719, made this method effectively useless. Removing it was a zero-diff change. I can look in future at containing the casts rather than removing them.

[saucecontrol]

The churn in this section is just taking care of this TODO

[saucecontrol]

Changing this to UnpackLow shows up as a regression in a few places, because movlhps is one byte smaller, but it enables other optimizations since unpcklpd takes a memory operand plus mask and embedded broadcast.

Vector128.Create(double, 1.0):

- vmovups xmm0, xmmword ptr [reloc @RWD00]- vmovlhps xmm0, xmm1, xmm0+ vunpcklpd xmm0, xmm1, qword ptr [reloc @RWD00] {1to2}

[tannergooding]

This should probably be peepholed back to vmovlhps if both are from register.

[saucecontrol]

I was thinking the same but would rather save that for a followup. llvm has a replacement list of equivalent instructions that have different sizes, and unpcklpd is on it, as are things like vpermilps, which is replaced by pshufd.

It's worth having a discussion about whether we'd also want to do replacements that switch between float and integer domains. I'll open an issue.

[saucecontrol]

I looked at this again, and there's actually only a size difference for legacy SSE encoding, so it's probably not worth special casing.

[saucecontrol]

In isolation, this change produced a small number of diffs and was mostly an improvement. A few regressions show up in the SPMI reports, but the overall impact is good, especially considering the places we can load a long to vector with movq

[saucecontrol]

It occurred to me the optimization to emit pinsrb/w for CreateScalarUnsafe was a bad idea because it creates a false dependency on the upper bits of the target reg. Removed that.

[tannergooding]

CC. @dotnet/jit-contrib, @EgorBo, @BruceForstall for secondary review

[saucecontrol]

Resolves#11626

This resolves a large number of TODOs around HWIntrinsic expansion involving scalar longs on x86.

The most significant change here is in promoting CreateScalar and ToScalar to be code generating intrinsics instead of converting them to other intrinsics at lowering. This was necessary in order to handle emitting movq for scalar long loads/stores but also unlocks several other optimizations since we can now allow CreateScalar and ToScalar to be contained and can specialize codegen depending on whether they end up loading/storing from/to memory or not. Some example improvements on x64:

Vector128.CreateScalar(ref float):

- vinsertps xmm0, xmm0, dword ptr [rbp+0x10], 14+ vmovss xmm0, dword ptr [rbp+0x10]

Vector128.CreateScalar(ref double):

- vxorps xmm0, xmm0, xmm0- vmovsd xmm1, qword ptr [rbp-0x08]- vmovsd xmm0, xmm0, xmm1+ vmovsd xmm0, qword ptr [rbp-0x08]

ref byte = Vector128<byte>.ToScalar():

- vmovd r9d, xmm3- mov byte ptr [r10], r9b+ vpextrb byte ptr [r10], xmm3, 0

Vector<byte>.ToScalar()

- vmovups ymm0, ymmword ptr [esp+0x04]- vmovd eax, xmm0- movzx eax, al+ movzx eax, byte ptr [esp+0x04]

And the less realistic, but still interesting
Sse.AddScalar(Vector128.CreateScalar(ref float), Vector128.CreateScalar(ref float)).ToScalar():

- xorps xmm0, xmm0- movss xmm1, dword ptr [rcx]- movss xmm0, xmm1- xorps xmm1, xmm1- movss xmm2, dword ptr [rdx]- movss xmm1, xmm2- addss xmm0, xmm1+ movss xmm0, dword ptr [rcx]+ addss xmm0, dword ptr [rdx]

This also removes some redundant casts for CreateScalar of small types. Previously, a zero-extending cast was inserted unconditionally and was sometimes removed by peephole opt on x64 but often wasn't.

Vector128.CreateScalar(short):

- movsx rax, dx- movzx rax, ax- movd xmm0, rax+ movzx rax, dx+ movd xmm0, eax

Vector128.CreateScalar(checked((byte)val)):

 cmp edx, 255
ja SHORT G_M000_IG04
mov eax, edx
- movzx rax, al- vmovd xmm0, rax+ vmovd xmm0, eax

Vector128.CreateScalar(ref sbyte):

- movsx rax, byte ptr [rdx]- movzx rax, al- vmovd xmm0, rax+ movzx rax, byte ptr [rdx]+ vmovd xmm0, eax

x86 diffs are much more significant, because of the newly-enabled intrinsic expansion:

Collection	Base size (bytes)	Diff size (bytes)	PerfScore in Diffs
benchmarks.run.windows.x86.checked.mch	7,149,204	-1,892	-2.17%
benchmarks.run_pgo.windows.x86.checked.mch	46,986,713	-738	+0.03%
benchmarks.run_tiered.windows.x86.checked.mch	9,470,045	-976	+0.11%
coreclr_tests.run.windows.x86.checked.mch	320,065,247	-205,564	-6.41%
libraries.crossgen2.windows.x86.checked.mch	31,314,339	-15,854	-4.11%
libraries.pmi.windows.x86.checked.mch	34,326,245	-14,416	-2.19%
libraries_tests.run.windows.x86.Release.mch	215,517,600	-55,366	-2.41%
libraries_tests_no_tiered_compilation.run.windows.x86.Release.mch	115,783,488	-80,576	-3.65%
realworld.run.windows.x86.checked.mch	9,587,950	-467	-0.45%

[saucecontrol]

This is ready for review.
cc @tannergooding

[saucecontrol]

This condition, added in #72719, made this method effectively useless. Removing it was a zero-diff change. I can look in future at containing the casts rather than removing them.

[saucecontrol]

The churn in this section is just taking care of this TODO

[saucecontrol]

Changing this to UnpackLow shows up as a regression in a few places, because movlhps is one byte smaller, but it enables other optimizations since unpcklpd takes a memory operand plus mask and embedded broadcast.

Vector128.Create(double, 1.0):

- vmovups xmm0, xmmword ptr [reloc @RWD00]- vmovlhps xmm0, xmm1, xmm0+ vunpcklpd xmm0, xmm1, qword ptr [reloc @RWD00] {1to2}

[tannergooding]

This should probably be peepholed back to vmovlhps if both are from register.

[saucecontrol]

I was thinking the same but would rather save that for a followup. llvm has a replacement list of equivalent instructions that have different sizes, and unpcklpd is on it, as are things like vpermilps, which is replaced by pshufd.

It's worth having a discussion about whether we'd also want to do replacements that switch between float and integer domains. I'll open an issue.

[saucecontrol]

I looked at this again, and there's actually only a size difference for legacy SSE encoding, so it's probably not worth special casing.

[saucecontrol]

In isolation, this change produced a small number of diffs and was mostly an improvement. A few regressions show up in the SPMI reports, but the overall impact is good, especially considering the places we can load a long to vector with movq

[saucecontrol]

It occurred to me the optimization to emit pinsrb/w for CreateScalarUnsafe was a bad idea because it creates a false dependency on the upper bits of the target reg. Removed that.

[tannergooding]

CC. @dotnet/jit-contrib, @EgorBo, @BruceForstall for secondary review

[saucecontrol]

Resolves#11626

This resolves a large number of TODOs around HWIntrinsic expansion involving scalar longs on x86.

The most significant change here is in promoting CreateScalar and ToScalar to be code generating intrinsics instead of converting them to other intrinsics at lowering. This was necessary in order to handle emitting movq for scalar long loads/stores but also unlocks several other optimizations since we can now allow CreateScalar and ToScalar to be contained and can specialize codegen depending on whether they end up loading/storing from/to memory or not. Some example improvements on x64:

Vector128.CreateScalar(ref float):

- vinsertps xmm0, xmm0, dword ptr [rbp+0x10], 14+ vmovss xmm0, dword ptr [rbp+0x10]

Vector128.CreateScalar(ref double):

- vxorps xmm0, xmm0, xmm0- vmovsd xmm1, qword ptr [rbp-0x08]- vmovsd xmm0, xmm0, xmm1+ vmovsd xmm0, qword ptr [rbp-0x08]

ref byte = Vector128<byte>.ToScalar():

- vmovd r9d, xmm3- mov byte ptr [r10], r9b+ vpextrb byte ptr [r10], xmm3, 0

Vector<byte>.ToScalar()

- vmovups ymm0, ymmword ptr [esp+0x04]- vmovd eax, xmm0- movzx eax, al+ movzx eax, byte ptr [esp+0x04]

And the less realistic, but still interesting
Sse.AddScalar(Vector128.CreateScalar(ref float), Vector128.CreateScalar(ref float)).ToScalar():

- xorps xmm0, xmm0- movss xmm1, dword ptr [rcx]- movss xmm0, xmm1- xorps xmm1, xmm1- movss xmm2, dword ptr [rdx]- movss xmm1, xmm2- addss xmm0, xmm1+ movss xmm0, dword ptr [rcx]+ addss xmm0, dword ptr [rdx]

This also removes some redundant casts for CreateScalar of small types. Previously, a zero-extending cast was inserted unconditionally and was sometimes removed by peephole opt on x64 but often wasn't.

Vector128.CreateScalar(short):

- movsx rax, dx- movzx rax, ax- movd xmm0, rax+ movzx rax, dx+ movd xmm0, eax

Vector128.CreateScalar(checked((byte)val)):

 cmp edx, 255
ja SHORT G_M000_IG04
mov eax, edx
- movzx rax, al- vmovd xmm0, rax+ vmovd xmm0, eax

Vector128.CreateScalar(ref sbyte):

- movsx rax, byte ptr [rdx]- movzx rax, al- vmovd xmm0, rax+ movzx rax, byte ptr [rdx]+ vmovd xmm0, eax

x86 diffs are much more significant, because of the newly-enabled intrinsic expansion:

Collection	Base size (bytes)	Diff size (bytes)	PerfScore in Diffs
benchmarks.run.windows.x86.checked.mch	7,149,204	-1,892	-2.17%
benchmarks.run_pgo.windows.x86.checked.mch	46,986,713	-738	+0.03%
benchmarks.run_tiered.windows.x86.checked.mch	9,470,045	-976	+0.11%
coreclr_tests.run.windows.x86.checked.mch	320,065,247	-205,564	-6.41%
libraries.crossgen2.windows.x86.checked.mch	31,314,339	-15,854	-4.11%
libraries.pmi.windows.x86.checked.mch	34,326,245	-14,416	-2.19%
libraries_tests.run.windows.x86.Release.mch	215,517,600	-55,366	-2.41%
libraries_tests_no_tiered_compilation.run.windows.x86.Release.mch	115,783,488	-80,576	-3.65%
realworld.run.windows.x86.checked.mch	9,587,950	-467	-0.45%

[saucecontrol]

This is ready for review.
cc @tannergooding

[saucecontrol]

This condition, added in #72719, made this method effectively useless. Removing it was a zero-diff change. I can look in future at containing the casts rather than removing them.

[saucecontrol]

The churn in this section is just taking care of this TODO

[saucecontrol]

Changing this to UnpackLow shows up as a regression in a few places, because movlhps is one byte smaller, but it enables other optimizations since unpcklpd takes a memory operand plus mask and embedded broadcast.

Vector128.Create(double, 1.0):

- vmovups xmm0, xmmword ptr [reloc @RWD00]- vmovlhps xmm0, xmm1, xmm0+ vunpcklpd xmm0, xmm1, qword ptr [reloc @RWD00] {1to2}

[tannergooding]

This should probably be peepholed back to vmovlhps if both are from register.

[saucecontrol]

I was thinking the same but would rather save that for a followup. llvm has a replacement list of equivalent instructions that have different sizes, and unpcklpd is on it, as are things like vpermilps, which is replaced by pshufd.

It's worth having a discussion about whether we'd also want to do replacements that switch between float and integer domains. I'll open an issue.

[saucecontrol]

I looked at this again, and there's actually only a size difference for legacy SSE encoding, so it's probably not worth special casing.

[saucecontrol]

In isolation, this change produced a small number of diffs and was mostly an improvement. A few regressions show up in the SPMI reports, but the overall impact is good, especially considering the places we can load a long to vector with movq

[saucecontrol]

It occurred to me the optimization to emit pinsrb/w for CreateScalarUnsafe was a bad idea because it creates a false dependency on the upper bits of the target reg. Removed that.

[tannergooding]

CC. @dotnet/jit-contrib, @EgorBo, @BruceForstall for secondary review

[saucecontrol]

Resolves#11626

This resolves a large number of TODOs around HWIntrinsic expansion involving scalar longs on x86.

The most significant change here is in promoting CreateScalar and ToScalar to be code generating intrinsics instead of converting them to other intrinsics at lowering. This was necessary in order to handle emitting movq for scalar long loads/stores but also unlocks several other optimizations since we can now allow CreateScalar and ToScalar to be contained and can specialize codegen depending on whether they end up loading/storing from/to memory or not. Some example improvements on x64:

Vector128.CreateScalar(ref float):

- vinsertps xmm0, xmm0, dword ptr [rbp+0x10], 14+ vmovss xmm0, dword ptr [rbp+0x10]

Vector128.CreateScalar(ref double):

- vxorps xmm0, xmm0, xmm0- vmovsd xmm1, qword ptr [rbp-0x08]- vmovsd xmm0, xmm0, xmm1+ vmovsd xmm0, qword ptr [rbp-0x08]

ref byte = Vector128<byte>.ToScalar():

- vmovd r9d, xmm3- mov byte ptr [r10], r9b+ vpextrb byte ptr [r10], xmm3, 0

Vector<byte>.ToScalar()

- vmovups ymm0, ymmword ptr [esp+0x04]- vmovd eax, xmm0- movzx eax, al+ movzx eax, byte ptr [esp+0x04]

And the less realistic, but still interesting
Sse.AddScalar(Vector128.CreateScalar(ref float), Vector128.CreateScalar(ref float)).ToScalar():

- xorps xmm0, xmm0- movss xmm1, dword ptr [rcx]- movss xmm0, xmm1- xorps xmm1, xmm1- movss xmm2, dword ptr [rdx]- movss xmm1, xmm2- addss xmm0, xmm1+ movss xmm0, dword ptr [rcx]+ addss xmm0, dword ptr [rdx]

This also removes some redundant casts for CreateScalar of small types. Previously, a zero-extending cast was inserted unconditionally and was sometimes removed by peephole opt on x64 but often wasn't.

Vector128.CreateScalar(short):

- movsx rax, dx- movzx rax, ax- movd xmm0, rax+ movzx rax, dx+ movd xmm0, eax

Vector128.CreateScalar(checked((byte)val)):

 cmp edx, 255
ja SHORT G_M000_IG04
mov eax, edx
- movzx rax, al- vmovd xmm0, rax+ vmovd xmm0, eax

Vector128.CreateScalar(ref sbyte):

- movsx rax, byte ptr [rdx]- movzx rax, al- vmovd xmm0, rax+ movzx rax, byte ptr [rdx]+ vmovd xmm0, eax

x86 diffs are much more significant, because of the newly-enabled intrinsic expansion:

Collection	Base size (bytes)	Diff size (bytes)	PerfScore in Diffs
benchmarks.run.windows.x86.checked.mch	7,149,204	-1,892	-2.17%
benchmarks.run_pgo.windows.x86.checked.mch	46,986,713	-738	+0.03%
benchmarks.run_tiered.windows.x86.checked.mch	9,470,045	-976	+0.11%
coreclr_tests.run.windows.x86.checked.mch	320,065,247	-205,564	-6.41%
libraries.crossgen2.windows.x86.checked.mch	31,314,339	-15,854	-4.11%
libraries.pmi.windows.x86.checked.mch	34,326,245	-14,416	-2.19%
libraries_tests.run.windows.x86.Release.mch	215,517,600	-55,366	-2.41%
libraries_tests_no_tiered_compilation.run.windows.x86.Release.mch	115,783,488	-80,576	-3.65%
realworld.run.windows.x86.checked.mch	9,587,950	-467	-0.45%

[saucecontrol]

This is ready for review.
cc @tannergooding

[saucecontrol]

This condition, added in #72719, made this method effectively useless. Removing it was a zero-diff change. I can look in future at containing the casts rather than removing them.

[saucecontrol]

The churn in this section is just taking care of this TODO

[saucecontrol]

Changing this to UnpackLow shows up as a regression in a few places, because movlhps is one byte smaller, but it enables other optimizations since unpcklpd takes a memory operand plus mask and embedded broadcast.

Vector128.Create(double, 1.0):

- vmovups xmm0, xmmword ptr [reloc @RWD00]- vmovlhps xmm0, xmm1, xmm0+ vunpcklpd xmm0, xmm1, qword ptr [reloc @RWD00] {1to2}

[tannergooding]

This should probably be peepholed back to vmovlhps if both are from register.

[saucecontrol]

I was thinking the same but would rather save that for a followup. llvm has a replacement list of equivalent instructions that have different sizes, and unpcklpd is on it, as are things like vpermilps, which is replaced by pshufd.

It's worth having a discussion about whether we'd also want to do replacements that switch between float and integer domains. I'll open an issue.

[saucecontrol]

I looked at this again, and there's actually only a size difference for legacy SSE encoding, so it's probably not worth special casing.

[saucecontrol]

In isolation, this change produced a small number of diffs and was mostly an improvement. A few regressions show up in the SPMI reports, but the overall impact is good, especially considering the places we can load a long to vector with movq

[saucecontrol]

It occurred to me the optimization to emit pinsrb/w for CreateScalarUnsafe was a bad idea because it creates a false dependency on the upper bits of the target reg. Removed that.

[tannergooding]

CC. @dotnet/jit-contrib, @EgorBo, @BruceForstall for secondary review

[saucecontrol]

Resolves#11626

This resolves a large number of TODOs around HWIntrinsic expansion involving scalar longs on x86.

The most significant change here is in promoting CreateScalar and ToScalar to be code generating intrinsics instead of converting them to other intrinsics at lowering. This was necessary in order to handle emitting movq for scalar long loads/stores but also unlocks several other optimizations since we can now allow CreateScalar and ToScalar to be contained and can specialize codegen depending on whether they end up loading/storing from/to memory or not. Some example improvements on x64:

Vector128.CreateScalar(ref float):

- vinsertps xmm0, xmm0, dword ptr [rbp+0x10], 14+ vmovss xmm0, dword ptr [rbp+0x10]

Vector128.CreateScalar(ref double):

- vxorps xmm0, xmm0, xmm0- vmovsd xmm1, qword ptr [rbp-0x08]- vmovsd xmm0, xmm0, xmm1+ vmovsd xmm0, qword ptr [rbp-0x08]

ref byte = Vector128<byte>.ToScalar():

- vmovd r9d, xmm3- mov byte ptr [r10], r9b+ vpextrb byte ptr [r10], xmm3, 0

Vector<byte>.ToScalar()

- vmovups ymm0, ymmword ptr [esp+0x04]- vmovd eax, xmm0- movzx eax, al+ movzx eax, byte ptr [esp+0x04]

And the less realistic, but still interesting
Sse.AddScalar(Vector128.CreateScalar(ref float), Vector128.CreateScalar(ref float)).ToScalar():

- xorps xmm0, xmm0- movss xmm1, dword ptr [rcx]- movss xmm0, xmm1- xorps xmm1, xmm1- movss xmm2, dword ptr [rdx]- movss xmm1, xmm2- addss xmm0, xmm1+ movss xmm0, dword ptr [rcx]+ addss xmm0, dword ptr [rdx]

This also removes some redundant casts for CreateScalar of small types. Previously, a zero-extending cast was inserted unconditionally and was sometimes removed by peephole opt on x64 but often wasn't.

Vector128.CreateScalar(short):

- movsx rax, dx- movzx rax, ax- movd xmm0, rax+ movzx rax, dx+ movd xmm0, eax

Vector128.CreateScalar(checked((byte)val)):

 cmp edx, 255
ja SHORT G_M000_IG04
mov eax, edx
- movzx rax, al- vmovd xmm0, rax+ vmovd xmm0, eax

Vector128.CreateScalar(ref sbyte):

- movsx rax, byte ptr [rdx]- movzx rax, al- vmovd xmm0, rax+ movzx rax, byte ptr [rdx]+ vmovd xmm0, eax

x86 diffs are much more significant, because of the newly-enabled intrinsic expansion:

Collection	Base size (bytes)	Diff size (bytes)	PerfScore in Diffs
benchmarks.run.windows.x86.checked.mch	7,149,204	-1,892	-2.17%
benchmarks.run_pgo.windows.x86.checked.mch	46,986,713	-738	+0.03%
benchmarks.run_tiered.windows.x86.checked.mch	9,470,045	-976	+0.11%
coreclr_tests.run.windows.x86.checked.mch	320,065,247	-205,564	-6.41%
libraries.crossgen2.windows.x86.checked.mch	31,314,339	-15,854	-4.11%
libraries.pmi.windows.x86.checked.mch	34,326,245	-14,416	-2.19%
libraries_tests.run.windows.x86.Release.mch	215,517,600	-55,366	-2.41%
libraries_tests_no_tiered_compilation.run.windows.x86.Release.mch	115,783,488	-80,576	-3.65%
realworld.run.windows.x86.checked.mch	9,587,950	-467	-0.45%

[saucecontrol]

This is ready for review.
cc @tannergooding

[saucecontrol]

This condition, added in #72719, made this method effectively useless. Removing it was a zero-diff change. I can look in future at containing the casts rather than removing them.

[saucecontrol]

The churn in this section is just taking care of this TODO

[saucecontrol]

Changing this to UnpackLow shows up as a regression in a few places, because movlhps is one byte smaller, but it enables other optimizations since unpcklpd takes a memory operand plus mask and embedded broadcast.

Vector128.Create(double, 1.0):

- vmovups xmm0, xmmword ptr [reloc @RWD00]- vmovlhps xmm0, xmm1, xmm0+ vunpcklpd xmm0, xmm1, qword ptr [reloc @RWD00] {1to2}

[tannergooding]

This should probably be peepholed back to vmovlhps if both are from register.

[saucecontrol]

I was thinking the same but would rather save that for a followup. llvm has a replacement list of equivalent instructions that have different sizes, and unpcklpd is on it, as are things like vpermilps, which is replaced by pshufd.

It's worth having a discussion about whether we'd also want to do replacements that switch between float and integer domains. I'll open an issue.

[saucecontrol]

I looked at this again, and there's actually only a size difference for legacy SSE encoding, so it's probably not worth special casing.

[saucecontrol]

In isolation, this change produced a small number of diffs and was mostly an improvement. A few regressions show up in the SPMI reports, but the overall impact is good, especially considering the places we can load a long to vector with movq

[saucecontrol]

It occurred to me the optimization to emit pinsrb/w for CreateScalarUnsafe was a bad idea because it creates a false dependency on the upper bits of the target reg. Removed that.

[tannergooding]

CC. @dotnet/jit-contrib, @EgorBo, @BruceForstall for secondary review

[saucecontrol]

Resolves#11626

This resolves a large number of TODOs around HWIntrinsic expansion involving scalar longs on x86.

The most significant change here is in promoting CreateScalar and ToScalar to be code generating intrinsics instead of converting them to other intrinsics at lowering. This was necessary in order to handle emitting movq for scalar long loads/stores but also unlocks several other optimizations since we can now allow CreateScalar and ToScalar to be contained and can specialize codegen depending on whether they end up loading/storing from/to memory or not. Some example improvements on x64:

Vector128.CreateScalar(ref float):

- vinsertps xmm0, xmm0, dword ptr [rbp+0x10], 14+ vmovss xmm0, dword ptr [rbp+0x10]

Vector128.CreateScalar(ref double):

- vxorps xmm0, xmm0, xmm0- vmovsd xmm1, qword ptr [rbp-0x08]- vmovsd xmm0, xmm0, xmm1+ vmovsd xmm0, qword ptr [rbp-0x08]

ref byte = Vector128<byte>.ToScalar():

- vmovd r9d, xmm3- mov byte ptr [r10], r9b+ vpextrb byte ptr [r10], xmm3, 0

Vector<byte>.ToScalar()

- vmovups ymm0, ymmword ptr [esp+0x04]- vmovd eax, xmm0- movzx eax, al+ movzx eax, byte ptr [esp+0x04]

And the less realistic, but still interesting
Sse.AddScalar(Vector128.CreateScalar(ref float), Vector128.CreateScalar(ref float)).ToScalar():

- xorps xmm0, xmm0- movss xmm1, dword ptr [rcx]- movss xmm0, xmm1- xorps xmm1, xmm1- movss xmm2, dword ptr [rdx]- movss xmm1, xmm2- addss xmm0, xmm1+ movss xmm0, dword ptr [rcx]+ addss xmm0, dword ptr [rdx]

This also removes some redundant casts for CreateScalar of small types. Previously, a zero-extending cast was inserted unconditionally and was sometimes removed by peephole opt on x64 but often wasn't.

Vector128.CreateScalar(short):

- movsx rax, dx- movzx rax, ax- movd xmm0, rax+ movzx rax, dx+ movd xmm0, eax

Vector128.CreateScalar(checked((byte)val)):

 cmp edx, 255
ja SHORT G_M000_IG04
mov eax, edx
- movzx rax, al- vmovd xmm0, rax+ vmovd xmm0, eax

Vector128.CreateScalar(ref sbyte):

- movsx rax, byte ptr [rdx]- movzx rax, al- vmovd xmm0, rax+ movzx rax, byte ptr [rdx]+ vmovd xmm0, eax

x86 diffs are much more significant, because of the newly-enabled intrinsic expansion:

Collection	Base size (bytes)	Diff size (bytes)	PerfScore in Diffs
benchmarks.run.windows.x86.checked.mch	7,149,204	-1,892	-2.17%
benchmarks.run_pgo.windows.x86.checked.mch	46,986,713	-738	+0.03%
benchmarks.run_tiered.windows.x86.checked.mch	9,470,045	-976	+0.11%
coreclr_tests.run.windows.x86.checked.mch	320,065,247	-205,564	-6.41%
libraries.crossgen2.windows.x86.checked.mch	31,314,339	-15,854	-4.11%
libraries.pmi.windows.x86.checked.mch	34,326,245	-14,416	-2.19%
libraries_tests.run.windows.x86.Release.mch	215,517,600	-55,366	-2.41%
libraries_tests_no_tiered_compilation.run.windows.x86.Release.mch	115,783,488	-80,576	-3.65%
realworld.run.windows.x86.checked.mch	9,587,950	-467	-0.45%

[saucecontrol]

This is ready for review.
cc @tannergooding

[saucecontrol]

This condition, added in #72719, made this method effectively useless. Removing it was a zero-diff change. I can look in future at containing the casts rather than removing them.

[saucecontrol]

The churn in this section is just taking care of this TODO

[saucecontrol]

Changing this to UnpackLow shows up as a regression in a few places, because movlhps is one byte smaller, but it enables other optimizations since unpcklpd takes a memory operand plus mask and embedded broadcast.

Vector128.Create(double, 1.0):

- vmovups xmm0, xmmword ptr [reloc @RWD00]- vmovlhps xmm0, xmm1, xmm0+ vunpcklpd xmm0, xmm1, qword ptr [reloc @RWD00] {1to2}

[tannergooding]

This should probably be peepholed back to vmovlhps if both are from register.

[saucecontrol]

I was thinking the same but would rather save that for a followup. llvm has a replacement list of equivalent instructions that have different sizes, and unpcklpd is on it, as are things like vpermilps, which is replaced by pshufd.

It's worth having a discussion about whether we'd also want to do replacements that switch between float and integer domains. I'll open an issue.

[saucecontrol]

I looked at this again, and there's actually only a size difference for legacy SSE encoding, so it's probably not worth special casing.

[saucecontrol]

In isolation, this change produced a small number of diffs and was mostly an improvement. A few regressions show up in the SPMI reports, but the overall impact is good, especially considering the places we can load a long to vector with movq

[saucecontrol]

It occurred to me the optimization to emit pinsrb/w for CreateScalarUnsafe was a bad idea because it creates a false dependency on the upper bits of the target reg. Removed that.

[tannergooding]

CC. @dotnet/jit-contrib, @EgorBo, @BruceForstall for secondary review

[saucecontrol]

Resolves#11626

This resolves a large number of TODOs around HWIntrinsic expansion involving scalar longs on x86.

The most significant change here is in promoting CreateScalar and ToScalar to be code generating intrinsics instead of converting them to other intrinsics at lowering. This was necessary in order to handle emitting movq for scalar long loads/stores but also unlocks several other optimizations since we can now allow CreateScalar and ToScalar to be contained and can specialize codegen depending on whether they end up loading/storing from/to memory or not. Some example improvements on x64:

Vector128.CreateScalar(ref float):

- vinsertps xmm0, xmm0, dword ptr [rbp+0x10], 14+ vmovss xmm0, dword ptr [rbp+0x10]

Vector128.CreateScalar(ref double):

- vxorps xmm0, xmm0, xmm0- vmovsd xmm1, qword ptr [rbp-0x08]- vmovsd xmm0, xmm0, xmm1+ vmovsd xmm0, qword ptr [rbp-0x08]

ref byte = Vector128<byte>.ToScalar():

- vmovd r9d, xmm3- mov byte ptr [r10], r9b+ vpextrb byte ptr [r10], xmm3, 0

Vector<byte>.ToScalar()

- vmovups ymm0, ymmword ptr [esp+0x04]- vmovd eax, xmm0- movzx eax, al+ movzx eax, byte ptr [esp+0x04]

And the less realistic, but still interesting
Sse.AddScalar(Vector128.CreateScalar(ref float), Vector128.CreateScalar(ref float)).ToScalar():

- xorps xmm0, xmm0- movss xmm1, dword ptr [rcx]- movss xmm0, xmm1- xorps xmm1, xmm1- movss xmm2, dword ptr [rdx]- movss xmm1, xmm2- addss xmm0, xmm1+ movss xmm0, dword ptr [rcx]+ addss xmm0, dword ptr [rdx]

This also removes some redundant casts for CreateScalar of small types. Previously, a zero-extending cast was inserted unconditionally and was sometimes removed by peephole opt on x64 but often wasn't.

Vector128.CreateScalar(short):

- movsx rax, dx- movzx rax, ax- movd xmm0, rax+ movzx rax, dx+ movd xmm0, eax

Vector128.CreateScalar(checked((byte)val)):

 cmp edx, 255
ja SHORT G_M000_IG04
mov eax, edx
- movzx rax, al- vmovd xmm0, rax+ vmovd xmm0, eax

Vector128.CreateScalar(ref sbyte):

- movsx rax, byte ptr [rdx]- movzx rax, al- vmovd xmm0, rax+ movzx rax, byte ptr [rdx]+ vmovd xmm0, eax

x86 diffs are much more significant, because of the newly-enabled intrinsic expansion:

Collection	Base size (bytes)	Diff size (bytes)	PerfScore in Diffs
benchmarks.run.windows.x86.checked.mch	7,149,204	-1,892	-2.17%
benchmarks.run_pgo.windows.x86.checked.mch	46,986,713	-738	+0.03%
benchmarks.run_tiered.windows.x86.checked.mch	9,470,045	-976	+0.11%
coreclr_tests.run.windows.x86.checked.mch	320,065,247	-205,564	-6.41%
libraries.crossgen2.windows.x86.checked.mch	31,314,339	-15,854	-4.11%
libraries.pmi.windows.x86.checked.mch	34,326,245	-14,416	-2.19%
libraries_tests.run.windows.x86.Release.mch	215,517,600	-55,366	-2.41%
libraries_tests_no_tiered_compilation.run.windows.x86.Release.mch	115,783,488	-80,576	-3.65%
realworld.run.windows.x86.checked.mch	9,587,950	-467	-0.45%

[saucecontrol]

This is ready for review.
cc @tannergooding

[saucecontrol]

This condition, added in #72719, made this method effectively useless. Removing it was a zero-diff change. I can look in future at containing the casts rather than removing them.

[saucecontrol]

The churn in this section is just taking care of this TODO

[saucecontrol]

Changing this to UnpackLow shows up as a regression in a few places, because movlhps is one byte smaller, but it enables other optimizations since unpcklpd takes a memory operand plus mask and embedded broadcast.

Vector128.Create(double, 1.0):

- vmovups xmm0, xmmword ptr [reloc @RWD00]- vmovlhps xmm0, xmm1, xmm0+ vunpcklpd xmm0, xmm1, qword ptr [reloc @RWD00] {1to2}

[tannergooding]

This should probably be peepholed back to vmovlhps if both are from register.

[saucecontrol]

I was thinking the same but would rather save that for a followup. llvm has a replacement list of equivalent instructions that have different sizes, and unpcklpd is on it, as are things like vpermilps, which is replaced by pshufd.

It's worth having a discussion about whether we'd also want to do replacements that switch between float and integer domains. I'll open an issue.

[saucecontrol]

I looked at this again, and there's actually only a size difference for legacy SSE encoding, so it's probably not worth special casing.

[saucecontrol]

In isolation, this change produced a small number of diffs and was mostly an improvement. A few regressions show up in the SPMI reports, but the overall impact is good, especially considering the places we can load a long to vector with movq

[saucecontrol]

It occurred to me the optimization to emit pinsrb/w for CreateScalarUnsafe was a bad idea because it creates a false dependency on the upper bits of the target reg. Removed that.

[tannergooding]

CC. @dotnet/jit-contrib, @EgorBo, @BruceForstall for secondary review

[saucecontrol]

Resolves#11626

This resolves a large number of TODOs around HWIntrinsic expansion involving scalar longs on x86.

The most significant change here is in promoting CreateScalar and ToScalar to be code generating intrinsics instead of converting them to other intrinsics at lowering. This was necessary in order to handle emitting movq for scalar long loads/stores but also unlocks several other optimizations since we can now allow CreateScalar and ToScalar to be contained and can specialize codegen depending on whether they end up loading/storing from/to memory or not. Some example improvements on x64:

Vector128.CreateScalar(ref float):

- vinsertps xmm0, xmm0, dword ptr [rbp+0x10], 14+ vmovss xmm0, dword ptr [rbp+0x10]

Vector128.CreateScalar(ref double):

- vxorps xmm0, xmm0, xmm0- vmovsd xmm1, qword ptr [rbp-0x08]- vmovsd xmm0, xmm0, xmm1+ vmovsd xmm0, qword ptr [rbp-0x08]

ref byte = Vector128<byte>.ToScalar():

- vmovd r9d, xmm3- mov byte ptr [r10], r9b+ vpextrb byte ptr [r10], xmm3, 0

Vector<byte>.ToScalar()

- vmovups ymm0, ymmword ptr [esp+0x04]- vmovd eax, xmm0- movzx eax, al+ movzx eax, byte ptr [esp+0x04]

And the less realistic, but still interesting
Sse.AddScalar(Vector128.CreateScalar(ref float), Vector128.CreateScalar(ref float)).ToScalar():

- xorps xmm0, xmm0- movss xmm1, dword ptr [rcx]- movss xmm0, xmm1- xorps xmm1, xmm1- movss xmm2, dword ptr [rdx]- movss xmm1, xmm2- addss xmm0, xmm1+ movss xmm0, dword ptr [rcx]+ addss xmm0, dword ptr [rdx]

This also removes some redundant casts for CreateScalar of small types. Previously, a zero-extending cast was inserted unconditionally and was sometimes removed by peephole opt on x64 but often wasn't.

Vector128.CreateScalar(short):

- movsx rax, dx- movzx rax, ax- movd xmm0, rax+ movzx rax, dx+ movd xmm0, eax

Vector128.CreateScalar(checked((byte)val)):

 cmp edx, 255
ja SHORT G_M000_IG04
mov eax, edx
- movzx rax, al- vmovd xmm0, rax+ vmovd xmm0, eax

Vector128.CreateScalar(ref sbyte):

- movsx rax, byte ptr [rdx]- movzx rax, al- vmovd xmm0, rax+ movzx rax, byte ptr [rdx]+ vmovd xmm0, eax

x86 diffs are much more significant, because of the newly-enabled intrinsic expansion:

Collection	Base size (bytes)	Diff size (bytes)	PerfScore in Diffs
benchmarks.run.windows.x86.checked.mch	7,149,204	-1,892	-2.17%
benchmarks.run_pgo.windows.x86.checked.mch	46,986,713	-738	+0.03%
benchmarks.run_tiered.windows.x86.checked.mch	9,470,045	-976	+0.11%
coreclr_tests.run.windows.x86.checked.mch	320,065,247	-205,564	-6.41%
libraries.crossgen2.windows.x86.checked.mch	31,314,339	-15,854	-4.11%
libraries.pmi.windows.x86.checked.mch	34,326,245	-14,416	-2.19%
libraries_tests.run.windows.x86.Release.mch	215,517,600	-55,366	-2.41%
libraries_tests_no_tiered_compilation.run.windows.x86.Release.mch	115,783,488	-80,576	-3.65%
realworld.run.windows.x86.checked.mch	9,587,950	-467	-0.45%

[saucecontrol]

This is ready for review.
cc @tannergooding

[saucecontrol]

This condition, added in #72719, made this method effectively useless. Removing it was a zero-diff change. I can look in future at containing the casts rather than removing them.

[saucecontrol]

The churn in this section is just taking care of this TODO

[saucecontrol]

Changing this to UnpackLow shows up as a regression in a few places, because movlhps is one byte smaller, but it enables other optimizations since unpcklpd takes a memory operand plus mask and embedded broadcast.

Vector128.Create(double, 1.0):

- vmovups xmm0, xmmword ptr [reloc @RWD00]- vmovlhps xmm0, xmm1, xmm0+ vunpcklpd xmm0, xmm1, qword ptr [reloc @RWD00] {1to2}

[tannergooding]

This should probably be peepholed back to vmovlhps if both are from register.

[saucecontrol]

I was thinking the same but would rather save that for a followup. llvm has a replacement list of equivalent instructions that have different sizes, and unpcklpd is on it, as are things like vpermilps, which is replaced by pshufd.

It's worth having a discussion about whether we'd also want to do replacements that switch between float and integer domains. I'll open an issue.

[saucecontrol]

I looked at this again, and there's actually only a size difference for legacy SSE encoding, so it's probably not worth special casing.

[saucecontrol]

In isolation, this change produced a small number of diffs and was mostly an improvement. A few regressions show up in the SPMI reports, but the overall impact is good, especially considering the places we can load a long to vector with movq

[saucecontrol]

It occurred to me the optimization to emit pinsrb/w for CreateScalarUnsafe was a bad idea because it creates a false dependency on the upper bits of the target reg. Removed that.

[tannergooding]

CC. @dotnet/jit-contrib, @EgorBo, @BruceForstall for secondary review