[Arm64] Use stp and str (SIMD) for stack prolog zeroing

[echesakov]

Currently, void CodeGen::genZeroInitFrame(int untrLclHi, int untrLclLo, regNumber initReg, bool* pInitRegZeroed)
inlines a zeroing loop for frames larger than 10 machine words (80 bytes on Arm64). The loop uses a wzr or xzr register and stp or str instructions and can write up to 16 bytes of zeros at once.

Following ideas in #32538 we can

1. zero-init a SIMD register qReg
2. use the register qReg instead of xzr with stp qReg, qReg, [mem] allowing to write up to 32 bytes of zeros to memory in one instruction.

We can also consider increasing the upper boundary (i.e. 10 machine words) to some larger number.

It seems that Clang/LLVM uses similar way for initializing stack allocated structs.
https://godbolt.org/z/8rKxvn

For example,

#include <string.h>

struct int32x4 
{
 int _1;
 int _2;
 int _3;
 int _4;
};

struct int32x8 
{
 int _1;
 int _2;
 int _3;
 int _4;
 int _5;
 int _6;
 int _7;
 int _8;
};

struct int32x16
{
 int _1;
 int _2;
 int _3;
 int _4;
 int _5;
 int _6;
 int _7;
 int _8;
 int _9;
 int _10;
 int _11;
 int _12;
 int _13;
 int _14;
 int _15;
 int _16;
};

void ZeroInt32x4(void* pDst, int cnt)
{
 int32x4 src = { };
 memcpy(pDst, &src, cnt);
}

void ZeroInt32x8(void* pDst, int cnt)
{
 int32x8 src = { };
 memcpy(pDst, &src, cnt);
}

void ZeroInt32x16(void* pDst, int cnt)
{
 int32x16 src = { };
 memcpy(pDst, &src, cnt);
}

would be compiled down to

ZeroInt32x4(void*, int): // @ZeroInt32x4(void*, int)
 sub sp, sp, #32 // =32
 stp x29, x30, [sp, #16] // 16-byte Folded Spill
 add x29, sp, #16 // =16
 sxtw x2, w1
 mov x1, sp
 stp xzr, xzr, [sp]
 bl memcpy
 ldp x29, x30, [sp, #16] // 16-byte Folded Reload
 add sp, sp, #32 // =32
 ret
ZeroInt32x8(void*, int): // @ZeroInt32x8(void*, int)
 sub sp, sp, #48 // =48
 stp x29, x30, [sp, #32] // 16-byte Folded Spill
 add x29, sp, #32 // =32
 movi v0.2d, #0000000000000000
 sxtw x2, w1
 mov x1, sp
 stp q0, q0, [sp]
 bl memcpy
 ldp x29, x30, [sp, #32] // 16-byte Folded Reload
 add sp, sp, #48 // =48
 ret
ZeroInt32x16(void*, int): // @ZeroInt32x16(void*, int)
 sub sp, sp, #80 // =80
 stp x29, x30, [sp, #64] // 16-byte Folded Spill
 add x29, sp, #64 // =64
 movi v0.2d, #0000000000000000
 sxtw x2, w1
 mov x1, sp
 stp q0, q0, [sp, #32]
 stp q0, q0, [sp]
 bl memcpy
 ldp x29, x30, [sp, #64] // 16-byte Folded Reload
 add sp, sp, #80 // =80
 ret

@dotnet/jit-contrib @TamarChristinaArm

[TamarChristinaArm]

Indeed, this would be a good optimization. You might want to consider treating this as a general inlined memset and memcopy like the C compilers do, where 0 is just an small optimization. This way you can use it for Array.Fill and Span.Fill for small sets as well. In either case here are some interesting cases for you to look at https://godbolt.org/z/cff7Ed

In particular look at how the compiler behaves when the number of bytes to copy is not a power of 2. We then just issue overlapping instructions. i.e. to fill 46 bytes, we use one stp of q to fill 32 bytes, and fill the remaining 14 bytes by doing another str q but moving the pointer 2 bytes back into the field we just filled giving the overlapping set.

#include <string.h>

void ZeroInt32x4(void* pDst, void* pSrc)
{
 memset(pDst, 0, 16);
}

void ZeroInt32x8(void* pDst, void* pSrc)
{
 memset(pDst, 0, 32);
}

void ZeroInt32x16(void* pDst, void* pSrc)
{
 memset(pDst, 0, 64);
}

void ZeroInt32x5(void* pDst, void* pSrc)
{
 memset(pDst, 0, 24);
}

void memsetCharUneven(void* pDst)
{
 memset(pDst, 8, 30);
}

void memsetCharUnevenComplex(void* pDst)
{
 memset(pDst, 'c', 30);
}

void memsetChar(void* pDst)
{
 memset(pDst, 'c', 32);
}

void memsetCharUnevenUnknown(void* pDst, char c)
{
 memset(pDst, c, 30);
}

void memsetCharUnknown(void* pDst, char c)
{
 memset(pDst, c, 30);
}

void memsetUneven(void* pDst, void* pSrc)
{
 memset(pDst, 0, 46);
}

void MemcpyInt32x16(void* pDst, void* pSrc)
{
 memcpy(pDst, pSrc, 64);
}

and it's assembly

ZeroInt32x4(void*, void*): // @ZeroInt32x4(void*, void*)
 stp xzr, xzr, [x0]
 ret
ZeroInt32x8(void*, void*): // @ZeroInt32x8(void*, void*)
 movi v0.2d, #0000000000000000
 stp q0, q0, [x0]
 ret
ZeroInt32x16(void*, void*): // @ZeroInt32x16(void*, void*)
 movi v0.2d, #0000000000000000
 stp q0, q0, [x0, #32]
 stp q0, q0, [x0]
 ret
ZeroInt32x5(void*, void*): // @ZeroInt32x5(void*, void*)
 stp xzr, xzr, [x0]
 str xzr, [x0, #16]
 ret
memsetCharUneven(void*): // @memsetCharUneven(void*)
 mov x8, #578721382704613384
 stur x8, [x0, #22]
 stp x8, x8, [x0, #8]
 str x8, [x0]
 ret
memsetCharUnevenComplex(void*): // @memsetCharUnevenComplex(void*)
 mov x8, #25443
 movk x8, #25443, lsl #16
 movk x8, #25443, lsl #32
 movk x8, #25443, lsl #48
 stur x8, [x0, #22]
 stp x8, x8, [x0, #8]
 str x8, [x0]
 ret
memsetChar(void*): // @memsetChar(void*)
 movi v0.16b, #99
 stp q0, q0, [x0]
 ret
memsetCharUnevenUnknown(void*, char): // @memsetCharUnevenUnknown(void*, char)
 and x8, x1, #0xff
 mov x9, #72340172838076673
 mul x8, x8, x9
 stur x8, [x0, #22]
 stp x8, x8, [x0, #8]
 str x8, [x0]
 ret
memsetCharUnknown(void*, char): // @memsetCharUnknown(void*, char)
 and x8, x1, #0xff
 mov x9, #72340172838076673
 mul x8, x8, x9
 stur x8, [x0, #22]
 stp x8, x8, [x0, #8]
 str x8, [x0]
 ret
memsetUneven(void*, void*): // @memsetUneven(void*, void*)
 movi v0.2d, #0000000000000000
 stur q0, [x0, #30]
 stp q0, q0, [x0]
 ret
MemcpyInt32x16(void*, void*): // @MemcpyInt32x16(void*, void*)
 ldp q1, q0, [x1, #32]
 ldp q3, q2, [x1]
 stp q1, q0, [x0, #32]
 stp q3, q2, [x0]
 ret

[TamarChristinaArm]

As an additional side note, the loop in

runtime/src/coreclr/src/jit/codegencommon.cpp

Line 6160 in 54906ea

loop:

is very tight, I'd recommend against emitting that but instead unroll the loop a couple of iterations. See section 4.4 of the optimization guide https://developer.arm.com/documentation/swog309707/a . If you do end up emitting such tight loops it's best to ensure the loop entry point is aligned to a 32byte boundary, see section 4.8. This will minimize the number of fetches that have to be done to execute the loop.

[echesakov]

@TamarChristinaArm Thank you for all the pointers! Let me read through the documentation you mentioned and I will come back with the algorithm before implementing.

[kunalspathak]

If you do end up emitting such tight loops it's best to ensure the loop entry point is aligned to a 32byte boundary, see section 4.8.

Thanks for pointing that out @TamarChristinaArm . As part of #43227, we are also working on aligning loop bodies to 32B boundary.

[TamarChristinaArm]

We can also consider increasing the upper boundary (i.e. 10 machine words) to some larger number.

@echesakovMSFT coming back to this, particularly for zeroing you have an additional option:

On AArch64 you have the data cache zero instructions DC ZVA (Can be found under system registers not Instructions).

Essentially using this you can clear large blocks of memory by VA, but it can be disabled so support for it should be detected at runtime. This can be done by reading the DCZID_EL0 register. This register will tell you whether it's enabled and the size of the block it's configured to clear.

The expectation is that most systems will be configured with a usable amount such as 64-bytes in one go.
Once you know this you can use dc zva, <addr-reg> to do the clear, which will allow you to inline clearings for much larger blocks before needing a function call.

[echesakov]

We can also consider increasing the upper boundary (i.e. 10 machine words) to some larger number.

@echesakovMSFT coming back to this, particularly for zeroing you have an additional option:

On AArch64 you have the data cache zero instructions DC ZVA (Can be found under system registers not Instructions).

Essentially using this you can clear large blocks of memory by VA, but it can be disabled so support for it should be detected at runtime. This can be done by reading the DCZID_EL0 register. This register will tell you whether it's enabled and the size of the block it's configured to clear.

The expectation is that most systems will be configured with a usable amount such as 64-bytes in one go.
Once you know this you can use dc zva, <addr-reg> to do the clear, which will allow you to inline clearings for much larger blocks before needing a function call.

Thank you for following up, Tamar! I've heard about this instruction but, for some reason, I was under (wrong) impression that it required EL1 or higher. Is there any alignment requirement for the address in <addr-reg>? My copy of reference doesn't say so.

[TamarChristinaArm]

Is there any alignment requirement for the address in ? My copy of reference doesn't say so.

There is, but it's a bit sneakily described. The operation works on an entire cache line. If the address is not aligned to the cache line it will silently align it (ignore the lower bits) and clear the entire cache line.

Which means it will clear data you didn't intend to. So there is an alignment constraint for what you probably want, but not for the actual use of the instruction (as in, you won't get an alignment fault).

This requirement makes it impossible to use in static compilers (though we do use it in AoR's memset https://github.com/ARM-software/optimized-routines/blob/0f4ae0c5b561de25acb10130fd5e473ec038f89d/string/aarch64/memset.S#L79) but for JITs it may still be useful.