JIT ARM64-SVE: Add AddAcross by a74nh · Pull Request #101674 · dotnet/runtime

a74nh · 2024-04-29T10:11:55Z

Contributes towards #99957

ghost · 2024-04-29T10:12:03Z

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

a74nh · 2024-04-29T10:14:32Z


 // IF_SVE_AI_3A
- theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_1BYTE, REG_V1, REG_P4, REG_V2,
+ theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_SCALABLE, REG_V1, REG_P4, REG_V2,


All the codegen changes:
For these instructions, arg2 (EA_1BYTE etc) is never used as the return value is dependent on the input type which is already specified in opt.
Switching arg2 to EA_SCALABLE means there is no need to write special hwinstrinsiccodegen code.
I've changed the bare minimal of instructions needed to make this patch work. There are quite a few more reduction like instructions - we should do those as we get to them in the API

a74nh · 2024-04-29T10:23:22Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


This file is a copy of _SveUnaryOpTestTemplate.template with conditional tests removed as they can't be used for reduction.

We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

a74nh · 2024-04-29T12:56:49Z

@kunalspathak @dotnet/arm64-contrib

kunalspathak · 2024-04-29T15:43:36Z

superpmi-* failures are #101685

kunalspathak

added some questions and minor changes.

kunalspathak · 2024-04-29T16:38:56Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

kunalspathak · 2024-04-29T16:41:50Z


 public static ushort AddWidening(ushort op1, byte op2) => (ushort)(op1 + op2);

+ public static ulong AddWidening(ulong op1, byte op2) => (ulong)(op1 + (ulong)op2);


do we also not need the following?
public static uint AddWidening(uint op1, byte op2) => (uint)(op1 + (uint)op2);

Not for the testing. Sve.AddAcross() always widens to 64bits regardless of the input.
helper.cs seems to be taking the approach of only adding helper functions when they are needed.

kunalspathak · 2024-04-29T16:43:55Z

+ return reduceOp(op1[0], op1[1]);
+ }
+
+ if (op1.Length % 2 != 0)


do we have input parameters that exercise this condition?

No. This would require an vector where the number of elements was not a power of 2.
I wasn't sure if there was a way of raising an exception in the testing. So instead, NaN would ensure the test failed.

kunalspathak

LGTM

kunalspathak · 2024-05-01T02:51:53Z

/ba-g Failure is #101721

* JIT ARM64-SVE: Add AddAcross * Remove enum changes * Fix SVE tests max vector size to 512bit * fix zip test cases --------- Co-authored-by: Kunal Pathak <Kunal.Pathak@microsoft.com>

a74nh · 2024-04-29T10:11:55Z

Contributes towards #99957

ghost · 2024-04-29T10:12:03Z

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

a74nh · 2024-04-29T10:14:32Z


 // IF_SVE_AI_3A
- theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_1BYTE, REG_V1, REG_P4, REG_V2,
+ theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_SCALABLE, REG_V1, REG_P4, REG_V2,


All the codegen changes:
For these instructions, arg2 (EA_1BYTE etc) is never used as the return value is dependent on the input type which is already specified in opt.
Switching arg2 to EA_SCALABLE means there is no need to write special hwinstrinsiccodegen code.
I've changed the bare minimal of instructions needed to make this patch work. There are quite a few more reduction like instructions - we should do those as we get to them in the API

a74nh · 2024-04-29T10:23:22Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


This file is a copy of _SveUnaryOpTestTemplate.template with conditional tests removed as they can't be used for reduction.

We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

a74nh · 2024-04-29T12:56:49Z

@kunalspathak @dotnet/arm64-contrib

kunalspathak · 2024-04-29T15:43:36Z

superpmi-* failures are #101685

kunalspathak

added some questions and minor changes.

kunalspathak · 2024-04-29T16:38:56Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

kunalspathak · 2024-04-29T16:41:50Z


 public static ushort AddWidening(ushort op1, byte op2) => (ushort)(op1 + op2);

+ public static ulong AddWidening(ulong op1, byte op2) => (ulong)(op1 + (ulong)op2);


do we also not need the following?
public static uint AddWidening(uint op1, byte op2) => (uint)(op1 + (uint)op2);

Not for the testing. Sve.AddAcross() always widens to 64bits regardless of the input.
helper.cs seems to be taking the approach of only adding helper functions when they are needed.

kunalspathak · 2024-04-29T16:43:55Z

+ return reduceOp(op1[0], op1[1]);
+ }
+
+ if (op1.Length % 2 != 0)


do we have input parameters that exercise this condition?

No. This would require an vector where the number of elements was not a power of 2.
I wasn't sure if there was a way of raising an exception in the testing. So instead, NaN would ensure the test failed.

kunalspathak

LGTM

kunalspathak · 2024-05-01T02:51:53Z

/ba-g Failure is #101721

* JIT ARM64-SVE: Add AddAcross * Remove enum changes * Fix SVE tests max vector size to 512bit * fix zip test cases --------- Co-authored-by: Kunal Pathak <Kunal.Pathak@microsoft.com>

a74nh · 2024-04-29T10:11:55Z

Contributes towards #99957

ghost · 2024-04-29T10:12:03Z

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

a74nh · 2024-04-29T10:14:32Z


 // IF_SVE_AI_3A
- theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_1BYTE, REG_V1, REG_P4, REG_V2,
+ theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_SCALABLE, REG_V1, REG_P4, REG_V2,


All the codegen changes:
For these instructions, arg2 (EA_1BYTE etc) is never used as the return value is dependent on the input type which is already specified in opt.
Switching arg2 to EA_SCALABLE means there is no need to write special hwinstrinsiccodegen code.
I've changed the bare minimal of instructions needed to make this patch work. There are quite a few more reduction like instructions - we should do those as we get to them in the API

a74nh · 2024-04-29T10:23:22Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


This file is a copy of _SveUnaryOpTestTemplate.template with conditional tests removed as they can't be used for reduction.

We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

a74nh · 2024-04-29T12:56:49Z

@kunalspathak @dotnet/arm64-contrib

kunalspathak · 2024-04-29T15:43:36Z

superpmi-* failures are #101685

kunalspathak

added some questions and minor changes.

kunalspathak · 2024-04-29T16:38:56Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

kunalspathak · 2024-04-29T16:41:50Z


 public static ushort AddWidening(ushort op1, byte op2) => (ushort)(op1 + op2);

+ public static ulong AddWidening(ulong op1, byte op2) => (ulong)(op1 + (ulong)op2);


do we also not need the following?
public static uint AddWidening(uint op1, byte op2) => (uint)(op1 + (uint)op2);

Not for the testing. Sve.AddAcross() always widens to 64bits regardless of the input.
helper.cs seems to be taking the approach of only adding helper functions when they are needed.

kunalspathak · 2024-04-29T16:43:55Z

+ return reduceOp(op1[0], op1[1]);
+ }
+
+ if (op1.Length % 2 != 0)


do we have input parameters that exercise this condition?

No. This would require an vector where the number of elements was not a power of 2.
I wasn't sure if there was a way of raising an exception in the testing. So instead, NaN would ensure the test failed.

kunalspathak

LGTM

kunalspathak · 2024-05-01T02:51:53Z

/ba-g Failure is #101721

* JIT ARM64-SVE: Add AddAcross * Remove enum changes * Fix SVE tests max vector size to 512bit * fix zip test cases --------- Co-authored-by: Kunal Pathak <Kunal.Pathak@microsoft.com>

a74nh · 2024-04-29T10:11:55Z

Contributes towards #99957

ghost · 2024-04-29T10:12:03Z

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

a74nh · 2024-04-29T10:14:32Z


 // IF_SVE_AI_3A
- theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_1BYTE, REG_V1, REG_P4, REG_V2,
+ theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_SCALABLE, REG_V1, REG_P4, REG_V2,


All the codegen changes:
For these instructions, arg2 (EA_1BYTE etc) is never used as the return value is dependent on the input type which is already specified in opt.
Switching arg2 to EA_SCALABLE means there is no need to write special hwinstrinsiccodegen code.
I've changed the bare minimal of instructions needed to make this patch work. There are quite a few more reduction like instructions - we should do those as we get to them in the API

a74nh · 2024-04-29T10:23:22Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


This file is a copy of _SveUnaryOpTestTemplate.template with conditional tests removed as they can't be used for reduction.

We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

a74nh · 2024-04-29T12:56:49Z

@kunalspathak @dotnet/arm64-contrib

kunalspathak · 2024-04-29T15:43:36Z

superpmi-* failures are #101685

kunalspathak

added some questions and minor changes.

kunalspathak · 2024-04-29T16:38:56Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

kunalspathak · 2024-04-29T16:41:50Z


 public static ushort AddWidening(ushort op1, byte op2) => (ushort)(op1 + op2);

+ public static ulong AddWidening(ulong op1, byte op2) => (ulong)(op1 + (ulong)op2);


do we also not need the following?
public static uint AddWidening(uint op1, byte op2) => (uint)(op1 + (uint)op2);

Not for the testing. Sve.AddAcross() always widens to 64bits regardless of the input.
helper.cs seems to be taking the approach of only adding helper functions when they are needed.

kunalspathak · 2024-04-29T16:43:55Z

+ return reduceOp(op1[0], op1[1]);
+ }
+
+ if (op1.Length % 2 != 0)


do we have input parameters that exercise this condition?

No. This would require an vector where the number of elements was not a power of 2.
I wasn't sure if there was a way of raising an exception in the testing. So instead, NaN would ensure the test failed.

kunalspathak

LGTM

kunalspathak · 2024-05-01T02:51:53Z

/ba-g Failure is #101721

* JIT ARM64-SVE: Add AddAcross * Remove enum changes * Fix SVE tests max vector size to 512bit * fix zip test cases --------- Co-authored-by: Kunal Pathak <Kunal.Pathak@microsoft.com>

a74nh · 2024-04-29T10:11:55Z

Contributes towards #99957

ghost · 2024-04-29T10:12:03Z

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

a74nh · 2024-04-29T10:14:32Z


 // IF_SVE_AI_3A
- theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_1BYTE, REG_V1, REG_P4, REG_V2,
+ theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_SCALABLE, REG_V1, REG_P4, REG_V2,


All the codegen changes:
For these instructions, arg2 (EA_1BYTE etc) is never used as the return value is dependent on the input type which is already specified in opt.
Switching arg2 to EA_SCALABLE means there is no need to write special hwinstrinsiccodegen code.
I've changed the bare minimal of instructions needed to make this patch work. There are quite a few more reduction like instructions - we should do those as we get to them in the API

a74nh · 2024-04-29T10:23:22Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


This file is a copy of _SveUnaryOpTestTemplate.template with conditional tests removed as they can't be used for reduction.

We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

a74nh · 2024-04-29T12:56:49Z

@kunalspathak @dotnet/arm64-contrib

kunalspathak · 2024-04-29T15:43:36Z

superpmi-* failures are #101685

kunalspathak

added some questions and minor changes.

kunalspathak · 2024-04-29T16:38:56Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

kunalspathak · 2024-04-29T16:41:50Z


 public static ushort AddWidening(ushort op1, byte op2) => (ushort)(op1 + op2);

+ public static ulong AddWidening(ulong op1, byte op2) => (ulong)(op1 + (ulong)op2);


do we also not need the following?
public static uint AddWidening(uint op1, byte op2) => (uint)(op1 + (uint)op2);

Not for the testing. Sve.AddAcross() always widens to 64bits regardless of the input.
helper.cs seems to be taking the approach of only adding helper functions when they are needed.

kunalspathak · 2024-04-29T16:43:55Z

+ return reduceOp(op1[0], op1[1]);
+ }
+
+ if (op1.Length % 2 != 0)


do we have input parameters that exercise this condition?

No. This would require an vector where the number of elements was not a power of 2.
I wasn't sure if there was a way of raising an exception in the testing. So instead, NaN would ensure the test failed.

kunalspathak

LGTM

kunalspathak · 2024-05-01T02:51:53Z

/ba-g Failure is #101721

* JIT ARM64-SVE: Add AddAcross * Remove enum changes * Fix SVE tests max vector size to 512bit * fix zip test cases --------- Co-authored-by: Kunal Pathak <Kunal.Pathak@microsoft.com>

a74nh · 2024-04-29T10:11:55Z

Contributes towards #99957

ghost · 2024-04-29T10:12:03Z

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

a74nh · 2024-04-29T10:14:32Z


 // IF_SVE_AI_3A
- theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_1BYTE, REG_V1, REG_P4, REG_V2,
+ theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_SCALABLE, REG_V1, REG_P4, REG_V2,


All the codegen changes:
For these instructions, arg2 (EA_1BYTE etc) is never used as the return value is dependent on the input type which is already specified in opt.
Switching arg2 to EA_SCALABLE means there is no need to write special hwinstrinsiccodegen code.
I've changed the bare minimal of instructions needed to make this patch work. There are quite a few more reduction like instructions - we should do those as we get to them in the API

a74nh · 2024-04-29T10:23:22Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


This file is a copy of _SveUnaryOpTestTemplate.template with conditional tests removed as they can't be used for reduction.

We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

a74nh · 2024-04-29T12:56:49Z

@kunalspathak @dotnet/arm64-contrib

kunalspathak · 2024-04-29T15:43:36Z

superpmi-* failures are #101685

kunalspathak

added some questions and minor changes.

kunalspathak · 2024-04-29T16:38:56Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

kunalspathak · 2024-04-29T16:41:50Z


 public static ushort AddWidening(ushort op1, byte op2) => (ushort)(op1 + op2);

+ public static ulong AddWidening(ulong op1, byte op2) => (ulong)(op1 + (ulong)op2);


do we also not need the following?
public static uint AddWidening(uint op1, byte op2) => (uint)(op1 + (uint)op2);

Not for the testing. Sve.AddAcross() always widens to 64bits regardless of the input.
helper.cs seems to be taking the approach of only adding helper functions when they are needed.

kunalspathak · 2024-04-29T16:43:55Z

+ return reduceOp(op1[0], op1[1]);
+ }
+
+ if (op1.Length % 2 != 0)


do we have input parameters that exercise this condition?

No. This would require an vector where the number of elements was not a power of 2.
I wasn't sure if there was a way of raising an exception in the testing. So instead, NaN would ensure the test failed.

kunalspathak

LGTM

kunalspathak · 2024-05-01T02:51:53Z

/ba-g Failure is #101721

* JIT ARM64-SVE: Add AddAcross * Remove enum changes * Fix SVE tests max vector size to 512bit * fix zip test cases --------- Co-authored-by: Kunal Pathak <Kunal.Pathak@microsoft.com>

a74nh · 2024-04-29T10:11:55Z

Contributes towards #99957

ghost · 2024-04-29T10:12:03Z

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

a74nh · 2024-04-29T10:14:32Z


 // IF_SVE_AI_3A
- theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_1BYTE, REG_V1, REG_P4, REG_V2,
+ theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_SCALABLE, REG_V1, REG_P4, REG_V2,


All the codegen changes:
For these instructions, arg2 (EA_1BYTE etc) is never used as the return value is dependent on the input type which is already specified in opt.
Switching arg2 to EA_SCALABLE means there is no need to write special hwinstrinsiccodegen code.
I've changed the bare minimal of instructions needed to make this patch work. There are quite a few more reduction like instructions - we should do those as we get to them in the API

a74nh · 2024-04-29T10:23:22Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


This file is a copy of _SveUnaryOpTestTemplate.template with conditional tests removed as they can't be used for reduction.

We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

a74nh · 2024-04-29T12:56:49Z

@kunalspathak @dotnet/arm64-contrib

kunalspathak · 2024-04-29T15:43:36Z

superpmi-* failures are #101685

kunalspathak

added some questions and minor changes.

kunalspathak · 2024-04-29T16:38:56Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

kunalspathak · 2024-04-29T16:41:50Z


 public static ushort AddWidening(ushort op1, byte op2) => (ushort)(op1 + op2);

+ public static ulong AddWidening(ulong op1, byte op2) => (ulong)(op1 + (ulong)op2);


do we also not need the following?
public static uint AddWidening(uint op1, byte op2) => (uint)(op1 + (uint)op2);

Not for the testing. Sve.AddAcross() always widens to 64bits regardless of the input.
helper.cs seems to be taking the approach of only adding helper functions when they are needed.

kunalspathak · 2024-04-29T16:43:55Z

+ return reduceOp(op1[0], op1[1]);
+ }
+
+ if (op1.Length % 2 != 0)


do we have input parameters that exercise this condition?

No. This would require an vector where the number of elements was not a power of 2.
I wasn't sure if there was a way of raising an exception in the testing. So instead, NaN would ensure the test failed.

kunalspathak

LGTM

kunalspathak · 2024-05-01T02:51:53Z

/ba-g Failure is #101721

* JIT ARM64-SVE: Add AddAcross * Remove enum changes * Fix SVE tests max vector size to 512bit * fix zip test cases --------- Co-authored-by: Kunal Pathak <Kunal.Pathak@microsoft.com>

a74nh · 2024-04-29T10:11:55Z

Contributes towards #99957

ghost · 2024-04-29T10:12:03Z

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

a74nh · 2024-04-29T10:14:32Z


 // IF_SVE_AI_3A
- theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_1BYTE, REG_V1, REG_P4, REG_V2,
+ theEmitter->emitIns_R_R_R(INS_sve_saddv, EA_SCALABLE, REG_V1, REG_P4, REG_V2,


All the codegen changes:
For these instructions, arg2 (EA_1BYTE etc) is never used as the return value is dependent on the input type which is already specified in opt.
Switching arg2 to EA_SCALABLE means there is no need to write special hwinstrinsiccodegen code.
I've changed the bare minimal of instructions needed to make this patch work. There are quite a few more reduction like instructions - we should do those as we get to them in the API

a74nh · 2024-04-29T10:23:22Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


This file is a copy of _SveUnaryOpTestTemplate.template with conditional tests removed as they can't be used for reduction.

We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

a74nh · 2024-04-29T12:56:49Z

@kunalspathak @dotnet/arm64-contrib

kunalspathak · 2024-04-29T15:43:36Z

superpmi-* failures are #101685

kunalspathak

added some questions and minor changes.

kunalspathak · 2024-04-29T16:38:56Z

@@ -0,0 +1,302 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+


We can do cndSel(mask, AddAcross(a), falseVal). The result of this would be:
If 0th lane was active in mask, it would have the result of AddAcross(a)
For all the other lanes, it will either have 0 (for active lanes) or falseVal for in-active lanes
If that makes sense, can you add those in your new template? This might not be super meaningful but want to make sure that we test the jit code paths at least.
cc: @tannergooding

kunalspathak · 2024-04-29T16:41:50Z


 public static ushort AddWidening(ushort op1, byte op2) => (ushort)(op1 + op2);

+ public static ulong AddWidening(ulong op1, byte op2) => (ulong)(op1 + (ulong)op2);


do we also not need the following?
public static uint AddWidening(uint op1, byte op2) => (uint)(op1 + (uint)op2);

Not for the testing. Sve.AddAcross() always widens to 64bits regardless of the input.
helper.cs seems to be taking the approach of only adding helper functions when they are needed.

kunalspathak · 2024-04-29T16:43:55Z

+ return reduceOp(op1[0], op1[1]);
+ }
+
+ if (op1.Length % 2 != 0)


do we have input parameters that exercise this condition?

No. This would require an vector where the number of elements was not a power of 2.
I wasn't sure if there was a way of raising an exception in the testing. So instead, NaN would ensure the test failed.

kunalspathak

LGTM

kunalspathak · 2024-05-01T02:51:53Z

/ba-g Failure is #101721

* JIT ARM64-SVE: Add AddAcross * Remove enum changes * Fix SVE tests max vector size to 512bit * fix zip test cases --------- Co-authored-by: Kunal Pathak <Kunal.Pathak@microsoft.com>

ghost added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI new-api-needs-documentation labels Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

dotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

a74nh force-pushed the sve_addacross_github branch 2 times, most recently from a0aaade to df9385bCompare April 29, 2024 11:17

JIT ARM64-SVE: Add AddAcross
a3d0161

a74nh force-pushed the sve_addacross_github branch from df9385b to a3d0161Compare April 29, 2024 12:22

a74nh commented Apr 29, 2024
View reviewed changes

Comment threadsrc/coreclr/jit/hwintrinsic.h Outdated

a74nh marked this pull request as ready for review April 29, 2024 12:56

kunalspathak added the arm-sve Work related to arm64 SVE/SVE2 support label Apr 29, 2024

kunalspathak suggested changes Apr 29, 2024
View reviewed changes

kunalspathak mentioned this pull request Apr 29, 2024
Arm64: Implement SVE APIs #99957
Closed

a74nh added 3 commits April 30, 2024 10:05

Remove enum changes
660ada3

Fix SVE tests max vector size to 512bit
b9fc078

Merge main
66b4faf

kunalspathak approved these changes Apr 30, 2024
View reviewed changes

This was referenced Apr 30, 2024
Dead lettering tests #101524
Closed
System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.SpanScalarDestination_SpecialValues fails #101721
Closed

fix zip test cases
22182a9

kunalspathak merged commit a700005 into dotnet:mainMay 1, 2024

a74nh deleted the sve_addacross_github branch May 1, 2024 08:11

a74nh mentioned this pull request May 1, 2024
ARM64-SVE: No obvious way to mask an Across function #101770
Closed

TIHan mentioned this pull request May 2, 2024
JIT: ARM64 Assertion failed 'isScalableVectorSize(size)'#101786
Closed

github-actionsBot locked and limited conversation to collaborators May 31, 2024

ghost added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI new-api-needs-documentation labels Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

dotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

a74nh force-pushed the sve_addacross_github branch 2 times, most recently from a0aaade to df9385bCompare April 29, 2024 11:17

JIT ARM64-SVE: Add AddAcross
a3d0161

a74nh force-pushed the sve_addacross_github branch from df9385b to a3d0161Compare April 29, 2024 12:22

a74nh commented Apr 29, 2024
View reviewed changes

Comment threadsrc/coreclr/jit/hwintrinsic.h Outdated

a74nh marked this pull request as ready for review April 29, 2024 12:56

kunalspathak added the arm-sve Work related to arm64 SVE/SVE2 support label Apr 29, 2024

kunalspathak suggested changes Apr 29, 2024
View reviewed changes

kunalspathak mentioned this pull request Apr 29, 2024
Arm64: Implement SVE APIs #99957
Closed

a74nh added 3 commits April 30, 2024 10:05

Remove enum changes
660ada3

Fix SVE tests max vector size to 512bit
b9fc078

Merge main
66b4faf

kunalspathak approved these changes Apr 30, 2024
View reviewed changes

This was referenced Apr 30, 2024
Dead lettering tests #101524
Closed
System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.SpanScalarDestination_SpecialValues fails #101721
Closed

fix zip test cases
22182a9

kunalspathak merged commit a700005 into dotnet:mainMay 1, 2024

a74nh deleted the sve_addacross_github branch May 1, 2024 08:11

a74nh mentioned this pull request May 1, 2024
ARM64-SVE: No obvious way to mask an Across function #101770
Closed

TIHan mentioned this pull request May 2, 2024
JIT: ARM64 Assertion failed 'isScalableVectorSize(size)'#101786
Closed

github-actionsBot locked and limited conversation to collaborators May 31, 2024

ghost added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI new-api-needs-documentation labels Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

dotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

a74nh force-pushed the sve_addacross_github branch 2 times, most recently from a0aaade to df9385bCompare April 29, 2024 11:17

JIT ARM64-SVE: Add AddAcross
a3d0161

a74nh force-pushed the sve_addacross_github branch from df9385b to a3d0161Compare April 29, 2024 12:22

a74nh commented Apr 29, 2024
View reviewed changes

Comment threadsrc/coreclr/jit/hwintrinsic.h Outdated

a74nh marked this pull request as ready for review April 29, 2024 12:56

kunalspathak added the arm-sve Work related to arm64 SVE/SVE2 support label Apr 29, 2024

kunalspathak suggested changes Apr 29, 2024
View reviewed changes

kunalspathak mentioned this pull request Apr 29, 2024
Arm64: Implement SVE APIs #99957
Closed

a74nh added 3 commits April 30, 2024 10:05

Remove enum changes
660ada3

Fix SVE tests max vector size to 512bit
b9fc078

Merge main
66b4faf

kunalspathak approved these changes Apr 30, 2024
View reviewed changes

This was referenced Apr 30, 2024
Dead lettering tests #101524
Closed
System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.SpanScalarDestination_SpecialValues fails #101721
Closed

fix zip test cases
22182a9

kunalspathak merged commit a700005 into dotnet:mainMay 1, 2024

a74nh deleted the sve_addacross_github branch May 1, 2024 08:11

a74nh mentioned this pull request May 1, 2024
ARM64-SVE: No obvious way to mask an Across function #101770
Closed

TIHan mentioned this pull request May 2, 2024
JIT: ARM64 Assertion failed 'isScalableVectorSize(size)'#101786
Closed

github-actionsBot locked and limited conversation to collaborators May 31, 2024

ghost added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI new-api-needs-documentation labels Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

dotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

a74nh force-pushed the sve_addacross_github branch 2 times, most recently from a0aaade to df9385bCompare April 29, 2024 11:17

JIT ARM64-SVE: Add AddAcross
a3d0161

a74nh force-pushed the sve_addacross_github branch from df9385b to a3d0161Compare April 29, 2024 12:22

a74nh commented Apr 29, 2024
View reviewed changes

Comment threadsrc/coreclr/jit/hwintrinsic.h Outdated

a74nh marked this pull request as ready for review April 29, 2024 12:56

kunalspathak added the arm-sve Work related to arm64 SVE/SVE2 support label Apr 29, 2024

kunalspathak suggested changes Apr 29, 2024
View reviewed changes

kunalspathak mentioned this pull request Apr 29, 2024
Arm64: Implement SVE APIs #99957
Closed

a74nh added 3 commits April 30, 2024 10:05

Remove enum changes
660ada3

Fix SVE tests max vector size to 512bit
b9fc078

Merge main
66b4faf

kunalspathak approved these changes Apr 30, 2024
View reviewed changes

This was referenced Apr 30, 2024
Dead lettering tests #101524
Closed
System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.SpanScalarDestination_SpecialValues fails #101721
Closed

fix zip test cases
22182a9

kunalspathak merged commit a700005 into dotnet:mainMay 1, 2024

a74nh deleted the sve_addacross_github branch May 1, 2024 08:11

a74nh mentioned this pull request May 1, 2024
ARM64-SVE: No obvious way to mask an Across function #101770
Closed

TIHan mentioned this pull request May 2, 2024
JIT: ARM64 Assertion failed 'isScalableVectorSize(size)'#101786
Closed

github-actionsBot locked and limited conversation to collaborators May 31, 2024

ghost added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI new-api-needs-documentation labels Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

dotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

a74nh force-pushed the sve_addacross_github branch 2 times, most recently from a0aaade to df9385bCompare April 29, 2024 11:17

JIT ARM64-SVE: Add AddAcross
a3d0161

a74nh force-pushed the sve_addacross_github branch from df9385b to a3d0161Compare April 29, 2024 12:22

a74nh commented Apr 29, 2024
View reviewed changes

Comment threadsrc/coreclr/jit/hwintrinsic.h Outdated

a74nh marked this pull request as ready for review April 29, 2024 12:56

kunalspathak added the arm-sve Work related to arm64 SVE/SVE2 support label Apr 29, 2024

kunalspathak suggested changes Apr 29, 2024
View reviewed changes

kunalspathak mentioned this pull request Apr 29, 2024
Arm64: Implement SVE APIs #99957
Closed

a74nh added 3 commits April 30, 2024 10:05

Remove enum changes
660ada3

Fix SVE tests max vector size to 512bit
b9fc078

Merge main
66b4faf

kunalspathak approved these changes Apr 30, 2024
View reviewed changes

This was referenced Apr 30, 2024
Dead lettering tests #101524
Closed
System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.SpanScalarDestination_SpecialValues fails #101721
Closed

fix zip test cases
22182a9

kunalspathak merged commit a700005 into dotnet:mainMay 1, 2024

a74nh deleted the sve_addacross_github branch May 1, 2024 08:11

a74nh mentioned this pull request May 1, 2024
ARM64-SVE: No obvious way to mask an Across function #101770
Closed

TIHan mentioned this pull request May 2, 2024
JIT: ARM64 Assertion failed 'isScalableVectorSize(size)'#101786
Closed

github-actionsBot locked and limited conversation to collaborators May 31, 2024

ghost added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI new-api-needs-documentation labels Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

dotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

a74nh force-pushed the sve_addacross_github branch 2 times, most recently from a0aaade to df9385bCompare April 29, 2024 11:17

JIT ARM64-SVE: Add AddAcross
a3d0161

a74nh force-pushed the sve_addacross_github branch from df9385b to a3d0161Compare April 29, 2024 12:22

a74nh commented Apr 29, 2024
View reviewed changes

Comment threadsrc/coreclr/jit/hwintrinsic.h Outdated

a74nh marked this pull request as ready for review April 29, 2024 12:56

kunalspathak added the arm-sve Work related to arm64 SVE/SVE2 support label Apr 29, 2024

kunalspathak suggested changes Apr 29, 2024
View reviewed changes

kunalspathak mentioned this pull request Apr 29, 2024
Arm64: Implement SVE APIs #99957
Closed

a74nh added 3 commits April 30, 2024 10:05

Remove enum changes
660ada3

Fix SVE tests max vector size to 512bit
b9fc078

Merge main
66b4faf

kunalspathak approved these changes Apr 30, 2024
View reviewed changes

This was referenced Apr 30, 2024
Dead lettering tests #101524
Closed
System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.SpanScalarDestination_SpecialValues fails #101721
Closed

fix zip test cases
22182a9

kunalspathak merged commit a700005 into dotnet:mainMay 1, 2024

a74nh deleted the sve_addacross_github branch May 1, 2024 08:11

a74nh mentioned this pull request May 1, 2024
ARM64-SVE: No obvious way to mask an Across function #101770
Closed

TIHan mentioned this pull request May 2, 2024
JIT: ARM64 Assertion failed 'isScalableVectorSize(size)'#101786
Closed

github-actionsBot locked and limited conversation to collaborators May 31, 2024

ghost added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI new-api-needs-documentation labels Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

dotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

a74nh force-pushed the sve_addacross_github branch 2 times, most recently from a0aaade to df9385bCompare April 29, 2024 11:17

JIT ARM64-SVE: Add AddAcross
a3d0161

a74nh force-pushed the sve_addacross_github branch from df9385b to a3d0161Compare April 29, 2024 12:22

a74nh commented Apr 29, 2024
View reviewed changes

Comment threadsrc/coreclr/jit/hwintrinsic.h Outdated

a74nh marked this pull request as ready for review April 29, 2024 12:56

kunalspathak added the arm-sve Work related to arm64 SVE/SVE2 support label Apr 29, 2024

kunalspathak suggested changes Apr 29, 2024
View reviewed changes

kunalspathak mentioned this pull request Apr 29, 2024
Arm64: Implement SVE APIs #99957
Closed

a74nh added 3 commits April 30, 2024 10:05

Remove enum changes
660ada3

Fix SVE tests max vector size to 512bit
b9fc078

Merge main
66b4faf

kunalspathak approved these changes Apr 30, 2024
View reviewed changes

This was referenced Apr 30, 2024
Dead lettering tests #101524
Closed
System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.SpanScalarDestination_SpecialValues fails #101721
Closed

fix zip test cases
22182a9

kunalspathak merged commit a700005 into dotnet:mainMay 1, 2024

a74nh deleted the sve_addacross_github branch May 1, 2024 08:11

a74nh mentioned this pull request May 1, 2024
ARM64-SVE: No obvious way to mask an Across function #101770
Closed

TIHan mentioned this pull request May 2, 2024
JIT: ARM64 Assertion failed 'isScalableVectorSize(size)'#101786
Closed

github-actionsBot locked and limited conversation to collaborators May 31, 2024

ghost added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI new-api-needs-documentation labels Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

dotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label Apr 29, 2024

a74nh commented Apr 29, 2024
View reviewed changes

a74nh force-pushed the sve_addacross_github branch 2 times, most recently from a0aaade to df9385bCompare April 29, 2024 11:17

JIT ARM64-SVE: Add AddAcross
a3d0161

a74nh force-pushed the sve_addacross_github branch from df9385b to a3d0161Compare April 29, 2024 12:22

a74nh commented Apr 29, 2024
View reviewed changes

Comment threadsrc/coreclr/jit/hwintrinsic.h Outdated

a74nh marked this pull request as ready for review April 29, 2024 12:56

kunalspathak added the arm-sve Work related to arm64 SVE/SVE2 support label Apr 29, 2024

kunalspathak suggested changes Apr 29, 2024
View reviewed changes

kunalspathak mentioned this pull request Apr 29, 2024
Arm64: Implement SVE APIs #99957
Closed

a74nh added 3 commits April 30, 2024 10:05

Remove enum changes
660ada3

Fix SVE tests max vector size to 512bit
b9fc078

Merge main
66b4faf

kunalspathak approved these changes Apr 30, 2024
View reviewed changes

This was referenced Apr 30, 2024
Dead lettering tests #101524
Closed
System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.SpanScalarDestination_SpecialValues fails #101721
Closed

fix zip test cases
22182a9

kunalspathak merged commit a700005 into dotnet:mainMay 1, 2024

a74nh deleted the sve_addacross_github branch May 1, 2024 08:11

a74nh mentioned this pull request May 1, 2024
ARM64-SVE: No obvious way to mask an Across function #101770
Closed

TIHan mentioned this pull request May 2, 2024
JIT: ARM64 Assertion failed 'isScalableVectorSize(size)'#101786
Closed

github-actionsBot locked and limited conversation to collaborators May 31, 2024

		@@ -0,0 +1,302 @@
		// Licensed to the .NET Foundation under one or more agreements.
		// The .NET Foundation licenses this file to you under the MIT license.


		public static ushort AddWidening(ushort op1, byte op2) => (ushort)(op1 + op2);

		public static ulong AddWidening(ulong op1, byte op2) => (ulong)(op1 + (ulong)op2);

Uh oh!

Conversation

a74nh commented Apr 29, 2024

Uh oh!

ghost commented Apr 29, 2024

Uh oh!

a74nhApr 29, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

a74nh commented Apr 29, 2024

Uh oh!

kunalspathak commented Apr 29, 2024

Uh oh!

kunalspathak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kunalspathak left a comment

Choose a reason for hiding this comment

Uh oh!

kunalspathak commented May 1, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

a74nh commented Apr 29, 2024

Uh oh!

ghost commented Apr 29, 2024

Uh oh!

a74nhApr 29, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

a74nh commented Apr 29, 2024

Uh oh!

kunalspathak commented Apr 29, 2024

Uh oh!

kunalspathak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

a74nhApr 29, 2024•
edited
Loading

a74nhApr 29, 2024•
edited
Loading

a74nhApr 29, 2024•
edited
Loading